Jeffers was platted by the Inter-State Land Company on September 19, 1899. It was incorporated as a city a few days later, on September 28, 1899. In its first election, the following men were put into office: President, L. P. Dustin; recorder, Lewis E. Streater; along with trustees, A. W. Binger, A. A. Faust, and C. G. Fredricson.
The town was named for land-owner George Jeffers. A post office has been in operation at Jeffers since 1900.
The Jeffers Petroglyphs, a site listed on the National Register of Historic Places, is a series of Native American stone carvings dating from before European settlement. They are located several miles northeast of the town.
In the center of the city runs a north-south street named "Courthouse Ave". It is rumored that due to its proximity in the center of Cottonwood County, Jeffers was expected to become the county seat of Cottonwood County. Therefore, they chose to name the street in anticipation of a courthouse being built. However, Windom, a city 18 miles to the south, was named the county seat and housed the courthouse.