It broke out at about 6 a.m. Wednesday at a home on Madison Avenue.
Fire officials say several people were trapped on the third floor.
A ladder was sent up to rescue them. Three of the victims, including the infant, suffered from smoke inhalation.
There is no word on what caused the fire.
Investigators are looking into what started the fire.