Self-XSS is a social engineering attack used to gain control of victims' web accounts. In a self-XSS attack, the victim of the attack unknowingly runs malicious code in their own web browser, thus exposing it to the attacker.
Self-XSS operates by tricking users into copying and pasting malicious content into their browsers' web developer console. Usually, the attacker posts a message that says by copying and running certain code, the user will be able to hack another user's account. In fact, the code allows the attacker to hijack the victim's account.
The "self" part of the name comes from the fact that the user is attacking themself. The "XSS" part of the name comes from the abbreviation for cross-site scripting, because both attacks result in malicious code running on a legitimate site. However, the attacks don't have much else in common, because XSS is an attack against the website itself (which users cannot protect themselves against but can be fixed by the site operator making their site more secure), whereas self-XSS is a social engineering attack against the user (which savvy users can protect themselves against but the site operator cannot do anything about it).