- cross-posted to:
- opensource@lemmy.ml
- privacy@programming.dev
- privacy@lemmy.ml
- cross-posted to:
- opensource@lemmy.ml
- privacy@programming.dev
- privacy@lemmy.ml
Hi everyone,
I am the original author of Searx. I started Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control.
Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches that index entirely on your own infrastructure.
Hister can automatically index pages through its Firefox and Chrome extensions. It can also watch local directories, import browser history and bookmarks, index individual URLs, and crawl complete documentation sites.
The feature I find most useful is offline previews. Hister stores the readable content and HTML of indexed pages locally. You can open a result in a clean and sanitized preview beside the search results without visiting the original website again.
Some other features:
- Full text search across web pages, PDFs, docx files, Markdown, OrgMode and text files
- Phrase searches, field filters, date filters, wildcards, negation, aliases, labels, facets, and result priorities
- Optional semantic search using an embeddings endpoint you configure
- Persistent website crawls
- Imports from browser history, Linkwarden, Karakeep, Shaarli, Wallabag, and Linkding
- Web, terminal, command line, HTTP API, and MCP interfaces
- SQLite and PostgreSQL support, plus optional multiple user hosting
Hister cannot replace a global search engine (yet) for subjects you have never encountered because it only searches what you have indexed. My workflow is to search Hister first, then use its shortcut to fall back to traditional search when I need broader web results.
The project is free software under the AGPLv3+ license. It can be installed as a standalone binary or with Docker.
Project: https://github.com/asciimoo/hister
Website and documentation: https://hister.org/
Small read-only demo: https://demo.hister.org/
I’d appreciate feedback, questions, and suggestions as well as joining our growing community.
AI disclosure: AI assisted contributions are not strictly prohibited, but all contributions should be made by humans. More details: https://github.com/asciimoo/hister/blob/master/CONTRIBUTING.md#ai-policy
Removed by mod
The summary has numerous inaccuracies. Most importantly: it is pretty easy to delete content by topic or age. The
hister deletecommand can accept a search query to remove only matched documents. The same is true on the web UI “actions -> remove all matching documents”. You can quickly filter by age, simply queryupdated:>365d. Combine it with URLs, labels, domains or phrases.Excellent - thanks for clearing that up.
Is there a TTL / max database size per user setting? Say I have 4 users using the server; can I allocate a hard limit of 10GB per user, with 180 day retention rules?
Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?
I would like to further recommend Hister but your documentation is somewhat confusing at first blush.
This is one of the problems with relying on AI… it can produce an overwhelming amount of content with errors and inaccuracies throughout. If you don’t review and know the content yourself, you won’t know what it got wrong.
It’s genuinely rude to lazily use AI to produce such a large babble of details and then ask someone else to review it for errors when you haven’t reviewed it yourself. I know you didn’t mean it that way but that’s nonetheless the result. People are going to read your comment and be misled about this project all because they assume AI is accurate and you didn’t review its results.
Edit: Sheesh… I hadn’t even got to your shitty comments that followed. You use AI to make a low effort but highly verbose post and then get mad that the repo author won’t review it in detail for you when you can’t be arsed with reading the docs yourself.
Yep, and this induces anyone who wants to review the resulting slop to need to turn to AI as well to even deal with the amount (or dismiss as a whole).
AI is a viral infection on software development.
This is my final reply in this thread. The developer has said their piece, and I have said mine. Now you’ve waded in - so let me set the record straight.
I am a developer. I had genuine interest in this project. I read the Hister documentation and inspected parts of the repository because the documentation did not clearly answer several basic questions I had:
How SQLite, Bleve, and stored HTML relate.
Whether TTL or storage quotas exist.
How browser-history deletion affects stored data.
How previews differ from a real web archive.
What multi-user isolation actually covers.
Yes, I used AI to assemble a plain-language summary and labelled it accordingly. Not everyone keeps the Hister codebase in their head, not everyone talks in code review and if I had these questions, I’m willing to bet others did too. The AI wrote for a lay audience because I didn’t ask it to do QA, I asked it to ELI-5.
The summary contained errors. Fine. That’s AI for you. However, if neither I nor the AI could find clear answers after cloning the repo, that supports my point about opacity.
At no point did I request a line-by-line audit. “Points 2 and 5 are wrong” would have answered the question.
Declining would also have been reasonable. Hell, side stepping it would have been fine too. Instead the dev decided to note the inaccuracies and rudely brush them off.
Both you and the dev seem to be under the impression !selfhosted is a one way distribution channel.
The developer came here, invited questions, then turned the raw prawn when questions arrived.
I didn’t go to their their Github. I didn’t abuse them. I genuinely wanted to know more about their project and share it, perhaps even work to help improve it.
They - and now you, ostensibly a happy supporter of Hister - came here.
Your claims about my effort and intent are assumptions followed by personal abuse.
Try and walk a mile in someone else’s shoes before calling them low effort and shitty next time.
As I wrote, you can simply automate deletion by document age. Schedule a delete event on each day with the desired retention time defined as a filter expression. Database size limit isn’t available yet.
No, sorry, I don’t have time to correct a copy of a multiple screens long AI prompt.
Which parts are confusing?
I am happy to narrow it further.
I took the time to read the documentation, ask ChatGPT to summarise what I found, and then reduced my follow-up to a simple request:
«Which of points 1–7 are materially wrong?»
That is not the same as asking you to audit “multiple screens” of AI output.
If the answer is “2 and 5 are incorrect”, or even “I do not have time to review it”, that is perfectly fine.
However, dismissing it as “a multiple screens long AI prompt” does not only not answer the question, it comes off as abrasive.
As for the documentation, the confusing parts are exactly those I listed: retention, lifecycle management, browser ingestion, storage limits, deletion, multi-user behaviour, and, most importantly, what Hister actually is and who it is for.
What’s disappointing is not that you disagreed with the AI summary. AIs are idiots.
It that after inviting questions and feedback, your response to a genuine attempt to understand the project is curt dismissal.
The inner workings of Hister may be obvious to you; they are not obvious to others.
The point is that you came here specifically to invite questions and feedback.
“TL;DR” does not encourage the sort of community engagement you ostensibly came here to seek.
I think it is more abrasive to copy/paste a poorly formatted LLM output instead of taking the time and summarizing it to a few sentences just as you did in your previous post.
That is not what happened.
I fed your GitHub repository to a clanker because the documentation did not answer my questions. I then shared its summary here.
You replied afterwards and said the summary was wrong. Fair enough. I then asked which specific points were wrong.
You could have answered, declined, or ignored the post.
Instead, you deigned only to dismiss the effort, then blamed me for objecting.
You also asked which parts were confusing, although my previous reply had already listed those issues.
You did not address them then, either.
A prospective user should not need ChatGPT, a cloned repository, and several follow-up questions to understand key functions.
You invited feedback. Your documentation remains unclear on several points, including issues beyond those I listed.
Your responses show that further feedback is not worth my time.
Feedback or questions is appropriate. We don’t need to come in abrasive, and you didn’t need chatgpt to ask a followup, so we don’t need to be antagonistic.