Searching for amulets

Nick Doty
24 February 2021

I love this concept of an "amulet", a short poem with a particular sha-256 hash characteristic of four or more sequential 8s.

I suppose that a "found amulet" is such a poem that isn't written or generated, but discovered in existing writing.

Let's read every line of poetry from the Project Gutenberg corpus. (Thanks, Allison, for maintaining these files.)

In [7]:
import json
f = open('gutenberg-poetry-v001.ndjson')
all_lines = []
for line in f.readlines():
    all_lines.append(json.loads(line.strip()))
In [9]:
text_lines = []
text_lines = [x['s'] for x in all_lines]
In [19]:
len(text_lines)
Out[19]:
3085117
In [46]:
modified_lines = []
for line in text_lines:
    modified_line = line.rstrip(',;:.')
    if modified_line != line:
        modified_lines.append(modified_line)
In [48]:
full_lines = text_lines + modified_lines
In [58]:
len(full_lines)
Out[58]:
4930626

Almost 5 million lines, a nice corpus to search in.

In [20]:
from hashlib import sha256
In [22]:
sha256(text_lines[0].encode('utf-8')).hexdigest()
Out[22]:
'29be6cb6d6621688c230a487890ea2857401e9e4c1a531226e95121f08364ec8'
In [49]:
amulets = []
uncommon = []
rare = []
epic = []
legendary = []
for line in full_lines:
    digest = sha256(line.encode('utf-8')).hexdigest()
    if '8888' in digest:
        amulets.append(line)
        if '88888' in digest:
            uncommon.append(line)
            if '888888' in digest:
                rare.append(line)
                if '8888888' in digest:
                    epic.append(line)
                    if '88888888' in digest:
                        legendary.append(line)
In [63]:
len(amulets)
Out[63]:
4299
In [50]:
len(amulets)/len(full_lines)
Out[50]:
0.0008718974020743005

There are many (over .08%) amulets that we've found, and this fraction seems consistent.

It seems very close to what my scratch paper math assumes is the likelihood of finding a particular 4-character sequence in a 64-character random sequence with a 16 letter alphabet (ignoring double amulets!): $ 16^{60} * 60 / 16^{64} $.

More generally, for a sequence of N length, the likelihood should be about: $ 64-N/16^N $

We shouldn't expect to find a legendary amulet in our 5 million line corpus, but if we had one ten times as big we probably would. (There are ~500 million tweets per day, though many are too many bytes to fit this definition.)

It doesn't feel very self-contained, but there is one epic amulet line, from an epic poem, no less:

In [59]:
epic
Out[59]:
['had grazed the top of the bone, for Polydamas had come up to him']

And some of the rare amulets are sweet, and more fitting. I like:

At the devil's booth are all things sold

and

What! your soul was pure and true;

Almost aphoristic.

In [56]:
rare
Out[56]:
['had grazed the top of the bone, for Polydamas had come up to him',
 'Te Deums to the night, invoking rest',
 'They, like the foliage, change their tint to brown;',
 'His charioteer drive to the hollow barks,',
 'Nor sweet Acrisian Danäe, from whom',
 'Tityus and Ixion too.',
 'May Hermes, herald of the dead, convey',
 'Then they bent theyr good yew bowes,',
 'given--at least in the present state of the critical art--of the',
 'What! your soul was pure and true;',
 'Five score thousand of Sarrazins they take',
 "At the devil's booth are all things sold",
 'A book I have, a friend gave',
 "At the devil's booth are all things sold",
 "They sing'd, and roasted o'er the burning coals",
 'A book I have, a friend gave',
 'The food that you gave me was pleasant and sweet',
 '_Cor._ Brother, thou art too young to comprehend',
 'Then came Sapphira,--she, poor soul',
 "At the devil's booth are all things sold",
 'May Hesperus in gladness pass before thee']

Alas, none of the 4,000+ found amulets include the word "amulet", although of course there could be other ways for a line to be “aware of its circumstances”.

In [62]:
for line in amulets:
    if 'amulet' in line:
        print(line)

There are too many common amulets to skim through briefly, but here's the few hundred uncommon ones, which I enjoyed.

In [65]:
uncommon
Out[65]:
['"I will send a Prophet to you,',
 'Just above the roots, he cut it,',
 'no little seed or leaf can we find.  Ah, good Fairy, let us be',
 'Nor Ismarus was wanting to the war,',
 'The rage of arms, and ratify the peace.',
 'Ther-to we wrecched wommen no-thing conne,',
 'On this dank soil thistles muster,',
 'A cow in a meadow shakes her bell',
 'As he which out of conscience',
 'Lo, which a misbelieve is hier!',
 'That he ne brekth the holinesse',
 'Nay, by this beard, whose hairs are white enough,',
 'Sharp as a flash of light,',
 'A painful night of mists and dreams;',
 'Sharp as a flash of light,',
 "So thou be shadow, here I make thee ghost,'",
 'She studies to divert, with fruitless pain,',
 'In that, by reason of the lack of light,',
 'Slain by mine hand ye all shall lie in dust!"',
 'Then we hushed our laughter, and asked her',
 'Me to love more, yet wish thee well much lesse.',
 'I may at last most gloriously uncloud',
 'Hath been the gain of death and dissolution.',
 'For singing, Fate has given sighs,',
 'Among their forests marvelled at for height.',
 'Among their forests marvelled at for height.',
 'To scowl upon his Maker, well from him',
 '"O Marco!" I replied, shine arguments',
 'To scowl upon his Maker, well from him',
 '"O Marco!" I replied, shine arguments',
 'Yes, Fancy, come, my Fairy love!',
 'Marvelling they stood, and watched the long grass swirled',
 'Make me laugh or let me go now, for long faces in the moonlight',
 'Sharp as a flash of light,',
 'She wanders in her mournful garden,',
 'He circled thus forever tracing out',
 'To part them, others stayed them, and the fight',
 'Those treasures in my breast for others dwell,',
 'And every kingdom hath a grave.',
 'Drawn up I was right off my feet,',
 'If forty such were burned.',
 'But I "over-write" the bushmen!  Well, I own without a doubt',
 '"I will send a Prophet to you,',
 'Just above the roots, he cut it,',
 'As upon the topmost spray',
 "Though Europe's Master Europe's Rebel banned",
 'Where her beloved Collatinus lies.',
 'Where her beloved Collatinus lies.',
 "Now Ted's garden's fine and cleanly,",
 "Telemachus' entertainment at Sparta, where Menelaus tells",
 'the nightingale of the greenwood, sings sweet in the first',
 'in the land of the Cephallenians. And now these wax',
 'Give over now.',
 'With no aid of other art',
 'had grazed the top of the bone, for Polydamas had come up to him',
 'That took the place of love.',
 'Hath wisdom! He is quit of works in bliss!',
 'The world is a shadow and life is a pain;',
 'I fear no glitter of pageantry,',
 'With myrtles wreathed in golden hair,',
 'Through its long vivid pulse-stirrings',
 'No man had ever heard a nightingale,',
 'The robe that hath no seam.',
 'Fortunate, Fridthjof to follow',
 'His nostrils are flaming, so glad is he.',
 'Cease, then, my friend! a moment cease,',
 'The old record saith, on whom',
 'Full of these salutary resolutions, he left London in July, 1739,',
 'Te Deums to the night, invoking rest',
 'Man then multiplied, and flourished,',
 'drama. His vaudeville was a lighter musical-dramatic genre,',
 'Could there the step of Happiness be traced.',
 'Those swindlers show the force of venal power,',
 'The awaiting murder darkly veiled--',
 '"Better to serve my purpose foul',
 'If that complete the story / to you I shall unfold,',
 'They, like the foliage, change their tint to brown;',
 'Would all its real fools were half as wise',
 'And should find this new, improved Destroyer',
 'How _acres_ descend, is in law-books displayed,',
 'Wilde Rosen überschlugen',
 'To scowl upon his Maker, well from him',
 'To scowl upon his Maker, well from him',
 'To scowl upon his Maker, well from him',
 'To scowl upon his Maker, well from him',
 '"O Marco!" I replied, shine arguments',
 '"O Marco!" I replied, shine arguments',
 '"O Marco!" I replied, shine arguments',
 'To scowl upon his Maker, well from him',
 "When will that dying murmur be suppress'd?",
 "Just think of Carolina's sage",
 "Just think of Carolina's sage",
 'For that sweet impulse of the coming Spring,',
 'Your La: humbly ever.',
 'Drawing near the Easter Sunday',
 'To scowl upon his Maker, well from him',
 'The martial son of Atreus by a shaft',
 "The wounds by my avenging bolts impress'd.",
 'His charioteer drive to the hollow barks,',
 'Nor sweet Acrisian Danäe, from whom',
 'Thus on they go; smooth glides the bark of pine,',
 "Except the moment that they crush't him;",
 'He overtook her in a wood;',
 'So the bells for Christmas ring,',
 "But if thou read'st my book unto the end,",
 'Until his strength gave out.',
 'God and his [blessed] angels grant that France',
 'Of viewless waves and tides invisible,',
 'I said, "This deed, sir, will you do?"',
 'Dost thou now lave thy feet and ankles white?',
 'When do the reasoning Powers decline?',
 "Some hid beneath the debris, some covered o'er with slime,",
 'For fierce within her, bitter hatred burned,',
 'The weeping of the chiefs to hear.',
 'And to his fair young darling cleave.',
 'Upon the princely hero gazed,',
 'Tityus and Ixion too.',
 'Thou art? Is this a boon so kindly given,',
 'squabbling with the husbands of Athens; insomuch that four of the',
 'Overcome the water-sorcerers,',
 'Thou the mighty red-capped hero,',
 'figure of a Nereid on his shield. The three sons of Æacus were',
 'But none of us all will fall or shirk.',
 'Yet my passions are compareless;',
 "Do adverse voices fall on the world's ear.",
 'May Hermes, herald of the dead, convey',
 'And blade in hand, their mazes wheel;',
 "And show'd--through all the melancholy flight--",
 "When thirsty on some river's muddy brink.",
 "A man's higher being is knowing, 122.",
 'The azure serpent from the cavern deep;',
 '“The more she sings her incantations, we',
 'Then they bent theyr good yew bowes,',
 "Can't buy clothes that seem to fit him;",
 "'Neath yellow hide of foster-wolf thence, mighty in his mirth,",
 'The heath, a lake of darkness, lay',
 'Born of a light that has passed away,',
 'Þy mendeȝ mou{n}teȝ not a myte,',
 'Her helpless body, once the prize of love,',
 '"It is what I carry my pictures in."',
 '"I will send a Prophet to you,',
 'Just above the roots, he cut it,',
 '"I was prepared to say a great deal. I had dressed up in my',
 'given--at least in the present state of the critical art--of the',
 'And in the warm hedge grew lush Eglantine,',
 'And I will call to mine.',
 '"I will send a Prophet to you,',
 'Just above the roots, he cut it,',
 'Just above the roots, he cut it,',
 'That unheard axes fell the woods,',
 '(COLERIDGE: _Kubla Khan._)',
 'Praise of Blank Verse," opening:',
 'They knew his worth his heart is of a kind,',
 'Tithonus remains in perpetual old age and grayness.',
 'That burns, and breaks its lamp, but is not lost.',
 'And he drove the shoals of fishes,',
 'Amid those sovereign glories of the skies,',
 'Mistake that small soft face to-night,',
 'Which leans over to the lane,',
 'Which made Thee once unbreathing--from the wrack',
 'In solitudes of twilight or of dawn,',
 'They knew his worth his heart is of a kind,',
 'They knew his worth his heart is of a kind,',
 'That life is worn! and when no rude sounds stir',
 'The far and the near',
 'What! your soul was pure and true;',
 "Have listen'd to the Drama's voice of truth,",
 'Of the old wide-open fireplace, where the backlog, all aglow',
 "Unmask'd, and unsex'd she presents to our view,",
 'Her helpless body, once the prize of love,',
 'Alas, found here a watery grave;',
 'court-martial convened, to try his offence.',
 'Which publishers would put their trust in;',
 'Sharpness and wittiness of Expression.',
 'Thou has revealed thyself--the same of yore,',
 'It cleft the realms of darkness, and',
 'They followed hounds down the Mourne End Hill.',
 'In this net of treason and mortal strait to lie.',
 'that abomination. _Odi profanum vulgus_, I hate your',
 'Wake snakes, _to get into trouble_.',
 "By all the mysteries that are breath'd,",
 'The phantom, Beauty, in a mist of tears;',
 'As upon the topmost spray',
 'Drawn up I was right off my feet,',
 'A day came when the frail year paused and said:',
 'Abdaldar ate, and he was satisfied:',
 "And love therefrom the twilight's coming greyness.",
 'Whispering, "Be of cheer."',
 'Give me another horse I cry,',
 "These to their sad catastrophe betrayed!'",
 'And oh! my heart is heavy, in this dull and darkened room,',
 'Though thou the billows smooth,',
 'Del hijo de Jimena',
 'Because you would have loved him, and then I',
 'Five score thousand of Sarrazins they take',
 'But at the flood arrived, "Farewell," quoth he',
 'And with Orlando who the palm divide',
 'Rogero, on whom his sorrows press and prey',
 'Made harsh, made keen with love that knows me not',
 'And brighter burned the Christmas flame',
 "For his own folly by himself aton'd",
 "For his own folly by himself aton'd",
 'A typhoon tempest follows',
 "Yet when 'tis day",
 "'O Mary, Mother, and is this thy goal",
 'EDITH comes forward',
 'I thought of Fragilion, that shy figure among the birch-trees',
 'I thought of Fragilion, that shy figure among the birch-trees',
 'And one for the little boy that lives in the lane',
 'For meal and maut she doesna want',
 'For in the leafy lane',
 "What though, from fortune's lavish bounty",
 "At the devil's booth are all things sold",
 'A book I have, a friend gave',
 "I see I ain't dean reet",
 "Outsings the valley-pigeon's lays",
 "At the devil's booth are all things sold",
 'Bjorn, all pale, stands the doorway nigh',
 "In French; and French Sir's Greek to me",
 'The Queen is dead. The splendour of her days',
 "They sing'd, and roasted o'er the burning coals",
 'Wise and foolish, great and small',
 'Soon whispered it to kind repose',
 'On those cool waters where we used to dwell',
 'Faintly suspire',
 'He might have bled for the Maccabees',
 "For his own folly by himself aton'd",
 "For his own folly by himself aton'd",
 "For his own folly by himself aton'd",
 'A book I have, a friend gave',
 'Already footed it like mad',
 'what was the fate of the war-wolves after the battle',
 'Thrice then he sprang impetuous to the deed',
 "Spurr'd through the portal flew the rapid steeds",
 "Spurr'd through the portal flew the rapid steeds",
 'And now, their foremost intercepted all',
 'The food that you gave me was pleasant and sweet',
 "'N' he's pertickler ill",
 'Piecemeal or in succession;--showed, I say',
 'I now can see with better eyes',
 'Though an immortal, she felt cruel pain',
 'Wise and foolish, great and small',
 'Along the road to Arcady',
 'Nor did their topic vary',
 'Gayly in rhyme--for that, said he',
 'Couched in thy brightness, dream of fields divine',
 'Li has a pigtail and a fan',
 'Along the road to Arcady',
 'Thus to the gathered people spake',
 'Back through the city gates he sped',
 'Each man that pleases reassumes his place',
 'Abiding an appointment made',
 'Molly in de Bramble-brier',
 'A psalm that lifts the fervent soul on high',
 'All for my love so true',
 'And my eternal king',
 'Chases the day, beholds thee watching there',
 '“Because the dead looked up and spoke',
 'With hopeless heart felt the swift hours go by',
 'Over the sea from Tenko',
 'And none the worse for rack or wear',
 'I now can see with better eyes',
 '"Ah, wipe thy tears; in war thy rival see',
 'The great gates creak and groan',
 "Thrown below hope, but rais'd 'bove fear",
 'With this vast music marching by',
 'He caused, the Knight turns back his reverent gaze',
 'Greatest and least',
 "From mountain-peaks, to tell the foe's approaches",
 '"Nay! I will not give my pearl-strung necklace',
 '_Cor._ Brother, thou art too young to comprehend',
 'Then came Sapphira,--she, poor soul',
 'absoluteness of the claim of each',
 "And then went pensively down the Avenue's calm",
 'Phantoms vex thy virgin sleep',
 'For all my ragged coat',
 'He strove to think the world was all a cheat',
 "At the devil's booth are all things sold",
 'We have brought news',
 "Forlorn, abandon'd, and unbless'd",
 'Is little May',
 'Then health and joy would follow, as they ought',
 'Chases the day, beholds thee watching there',
 'That Christ was standing there with me',
 'Publica el amarillo jaramago',
 'May Hesperus in gladness pass before thee']