Born the youngest of 23 children in Archibald Patch, Lackawanna County, Pennsylvania, he nonetheless matriculated at the University of Chicago. He became a St. Louis union leader of Warehousemen, when St. Louis was America's fifth largest entrepot because of its situation on the Mississippi and Missouri Rivers, and a major rail hub. The Warehousemen and Department Store Worker merged into United Retail, Wholesale and Department Store Employees of America, and that local union merged into Teamsters Local 688 in St. Louis. The International Brotherhood of Teamsters was the third of three international trade union vice presidencies he held. He was also a vice president of the teachers' union and the AF of L. He was a delegate to Democratic National Convention from Missouri in 1952. He was vice-president of the Alliance for Labor Action to promote social concerns and to organize the unorganized. He was a member of the American Civil Liberties Union and NAACP.
The St. Louis union was considered to be one of the most progressive in the United States. It initiated health care centers for members, vacation centers at Lake of the Ozarks, and militated for good pension plans for its members. Under Gibbons the Union researched and submitted plans for the desegregation of schools which was promoted by the editorial page of the St. Louis Post-Dispatch. Gibbons and Local 688 were also noted for their support of public housing, and were instrumental in the construction of the large Council Plaza housing development, which has been listed on the National Register of Historical Places since 2007.
For a time, Gibbons was widely considered to be the heir apparent to Jimmy Hoffa. But Gibbons' work and political stances landed him on the master list of Nixon political opponents. Nixon's Chief Counsel, Charles Colson, directed White House Counsel John Dean to initiate tax audits on Gibbons, but Dean did not follow through. Gibbons' opposition to the Vietnam War led to Hoffa moving to marginalize him. Hoffa supported the war, while Gibbons had been a founder of Labor for Peace, and had visited Hanoi. Another source of friction was Bobby Kennedy, who had hounded Hoffa, and whom Gibbons had befriended. While Gibbons remained head of the Teamsters in St. Louis, he was maneuvered out of posts in which he could influence policy.
The site of the original Sportsman's Park baseball stadium in St. Louis, now a neighborhood playground, was named "Harold J. Gibbons Field" for him.
Gibbons' papers are in the archives of Southern Illinois University-Edwardsville which he was instrumental in founding, because Illinois union members who wished to pursue higher education had to make exhausting commutes to attend university in Carbondale.