Botox in Patterson, NY.

The town was first settled around 1720 in The Oblong, which was a disputed area in southeastern Province of New York also claimed by the Connecticut Colony. The Oblong was a strip of land approximately 2.9 km wide between Dutchess County, New York, and Connecticut, ceded to New York in the 1731 Treaty of Dover. Between 1720 and 1776 a large number of mostly Connecticut families settled in the southern Oblong. They could not settle west of it because that land was privately owned by the Philipse Family. It had been granted a patent for virtually all of the remainder of the area of the future Putnam County.

The first such settlers in the Oblong were the Hayt family, who built a house at The Elm in 1720. Another early settler was Jacob Haviland, who settled Haviland Hollow in 1731. The first village in Putnam County, the hamlet of Patterson, was originally called Frederickstown. The eastern part of the future Putnam County was called Southeast Precinct (not the same as the current town of Southeast).

The Philipses were Loyalists during the Revolution and left the area. The state confiscated their land, selling it off. In 1788, the former Philipse portion of the Oblong was chartered as the Town of Southeast; the remainder of the region was chartered as the Town of Fredericktown in Dutchess County, New York. In 1795, Fredericktown township was split into four parts: 1. the Town of Carmel, 2. a part which was combined with the northern half of Southeast and became the Town of Franklin, which was renamed the Town of Patterson in 1808, 3. a part which was combined with the southern half of Southeast and became the new, much larger town of Southeast, and 4. the remnant of the town, which was the Town of Frederick for a while. It was renamed as the Town of Kent in 1817.