The de Birmingham family held the lordship of Birmingham in England for four hundred years and managed its growth from a small village into a thriving market town. They also helped invade Ireland and were rewarded with the Barony of Athenry. They were stripped of most of their lands in England by the notorious John Dudley, Duke of Northumberland, who held sway over Edward VI during the early 16th century.
A Norman baron called Ansculf de Picquigny (Ausculph de Penchengi) was granted many manors in the county of Warwickshire, and elsewhere, at the time of the Norman Conquest. They were centred upon Dudley Castle. Ansculf was observed by Sir William Dugdale (who mistakenly confused him with his son William) to have been a great man in the time of the conquest as evidenced by the extent of the lands granted to him, i.e., ten lordships in Berkshire, one in Middlesex, one in Oxfordshire, one in Huntingdonshire, one in Cambridgeshire, seven in Surrey, four in Northamptonshire, seven in Warwickshire, twenty in Buckinghamshire, twenty-five in Staffordshire and fourteen in Worcestershire.
Fulk Paganell was succeeded by his son Ralph Paganell who was in turn succeeded by his son Gervais Paganell. Gervais took part in the unsuccessful rebellion against Henry II in 1173. As a result, he died, his lands were confiscated by the crown and Dudley Castle demolished.
The flow of large parts of the Fitz-Ausculph estate to the Paganells lends itself to the theory that William Fitz Ausculph died with no surviving male heirs and so gave his lands to his daughter and her Paganell heirs.
According to a poem a later William de Birmingham was descended from the Ausculphs via a marriage to the Paganells. According to William Hutton's An History of Birmingham another William de Birmingham claimed in 1309 to have had ancestors who had the right to have a market in Birmingham before the Norman Conquest which would indicate that they were an old Anglo-Saxon family and not Norman.
Some sources claim the right to hold a market in Birmingham was granted to a Peter Fitz William, Steward of Dudley, and not to Peter de Birmingham who was known as Steward of Gervais Paganell. At this time people could be known by more than one name so it is probable that Peter de Birmingham and Peter Fitz William were one and the same person.
Fulk Paganell has been tenuously claimed to have had several children, one of whom is said to have been called William. It is feasible that William would have been given a small part of the Paganell lands, e.g., Birmingham, while his older brother Ralph received the rest. This could have led to William Paganell becoming known as William de Birmingham.
According to Hutton, Richard was the name of the owner of Birmingham at the time of the Norman Conquest. Richard is said to have survived and to have been known as Lord of the manor of Birmingham by way of Knight's service to William Fitz-Ausculf.
In the Domesday Book of 1086 there is a record of a Richard (no surname) occupying four hides at Birmingham which belonged to William Fitz-Ansculf of Dudley Castle. Richard may or may not have been the first of the "de Birminghams".
 William left Peter land of little value; Birmingham was one of the poorest manors in Warwickshire. There was little woodland and it was covered largely in scrub.
He applied to hold a cattle and food market every Thursday in the grounds of his "castle" and was granted the right by King Henry II in 1154 (some say 1166). He and his heirs were responsible for its jurisdiction. Outsiders were charged to come to the market, encouraging many merchants to live within Birmingham town and hence to pay Peter rents that far exceeded the land's agricultural value. Birmingham became the most successful market in the whole of England. From a population of 50 people in 1086 the town grew to 1500 by 1300.
In 1166 Peter is recorded to have owned a "castle" at Birmingham, to have been the Steward of Gervais Paganell and to have held nine Knight's fees by military service. The "castle" is thought to have been the Birmingham Manor House - a fortified stone manor house surrounded by a circular moat and a range of outbuildings and was probably built in the 12th century. It was rebuilt in the 13th century and remained there until the 18th century. The moat has now been filled in and the house is now the site of the Bull Ring.
Peter bore for his coat of arms:- azure, a bend lozenge, of five points, or.
 The de Birmingham family were instrumental in helping Strongbow in the Norman Invasion of Ireland. They were described as "the noble and warlike family of the Bremichams" and were rewarded much later with the Barony of Athenry, initially as lords of the manor. Both Sir William de Birmingham and one of his youngest sons, Robert de Birmingham, are listed as being among the Normans sent in Henry II's invasion in 1172. Robert was later styled the 1st Baron Athenry.Meyler de Bermingham took part in the invasion of Connacht in the 1230s and started to build the town of Athenry c. 1240.
 Sir William's eldest son.
 William was granted the right to hold a four-day fair starting on the eve of Ascension Day by Henry III in 1250. In 1251 permission was also given to hold a two-day fair beginning on the eve of the Feast of St John the Baptist. William supported Simon de Montford in the Second Barons' War against Henry III and to have died at the Battle of Evesham in 1265. The manor of Birmingham was confiscated by the King and given to Roger de Clifford. William was married to the daughter of Thomas de Astley. Her name in 1263 was recorded as Maud.
A poem has been written about this William called William de Birmingham. Its historical accuracy cannot be verified but it mentions William's claim to de descended from Ausculph via marriage to the Paganells.
 William took up his right to reclaim his father's land by way of paying a fine. In 1283 he strengthened his claims to land in Stockton Worcester, Shetford (Shutford), Oxon, Maidencoat (Maidencourt) Berks, Hoggeston Bucks and Christleton Cheshire. Maidencourt had come into the de Birmingham via a marriage to Sybil de Colville, who was the daughter of Alice de Colville who held the manor of Maidencourt in the early 13th century.
In 1327 William was summoned to Parliament. By this time Birmingham had overtaken the County Town of Warwick in terms of tax revenue. Willam's friend, Hugh Spencer, gave him custody of Dudley Castle
William's tomb lies in Birmingham's original church, St Martin in the Bull Ring, which was originally built by the de Birminghams in the 12th century.
 Married Joan and made a settlement of Kingston Bagpuize, Berks in 1340 with the advowson of the church. Remarried to Elizabeth. Leased the manor of Kingston Bagpuize to Peter Coke for life in 1367.
Acted as a Member of Parliament for Warwick during the 1350s and 1360s.
Abandoned the de Birmingham's traditional coat of arms and replaced it with:- partie per pale, indented, or, and gules.
Married Elizabeth but had no children.
 Upon Sir John's death, the manor of Birmingham was left to his widow, Elizabeth, to use as a dower. She remarried to Lord Clinton and they lived in the de Birmingham's manor house in Birmingham until Elizabeth's death in 1424.
The rest of Sir John's lands were left to his brother, Sir Thomas de Birmingham, who built his own castle at Worstone near Birmingham. Upon Elizabeth's death Thomas was to have inherited the manor of Birmingham. However, he died before her and the de Birmingham lands were left to Thomas' two granddaughters:-
who were both daughters of Thomas' only daughter, Elizabeth, who had been the wife of Thomas de la Roche.
Lady Ellen Ferrers' inheritance included the manor of Birmingham, which she eventually passed to a second cousin called William de Birmingham.
 Lady Ferrers quit her title in favour of a second cousin, William de Birmingham, who appears to have been a descendant of William de Birmingham, brother of Sir Fulk de Birmingham. He held the manor by military service in 1441 and died in 1479.
According to Sir William Dugdale's Antiquities of Warwickshire Edward was the last de Birmingham to hold the lordship of Birmingham. He was cheated out of it by John Dudley, who would later try to place Lady Jane Grey, his own daughter-in-law, on the throne, an offence for which he was later found guilty of treason and sentenced to death.
Dudley had asked to buy Birmingham from Edward. However, proud of his inheritance, Edward declined. Dudley was not a man to take no for answer though and devised a scheme to obtain Birmingham by dishonest means. A trap was laid for Edward and he was framed for highway robbery by Dudley's thugs. He was thrown in the Tower of London, tried and found guilty. However, Dudley being King Edward VI's closest adviser offered to get Edward a pardon from the King. The offer had a string attached, however, and in order to get the pardon, Edward had to hand Birmingham over to Dudley. Preferring poverty to death Edward did as Dudley demanded and in 1527 retired to obscurity, living on a token £40 a year offered as compensation. Edward had died by 1538 although his widow, Elizabeth, continued to live in Birmingham after his death.
The lordship of Birmingham was not returned to the de Birmingham family upon Dudley's death. It reverted to the Crown and then to the Marrow family.
Other land held by the de Birminghams does not appear to have been signed over to Dudley and there are records of the de Birmingham's ownership of it after 1527, e.g., Shutford, Oxon in 1544 which mentions a young William, son of Henry, son of William & Margaret de Birmingham.
However, this too appears to have been transferred away from them by Henry VIII, possibly under the advice of John Dudley, who was not executed until 1553.