I run a website that's in a similar situation, and it sounds like the author is close to the correct solution, but needs to take things one step further:
There needs to be a Single Point of Truth, and it should not be your little website.
BigCentralThing should be the official database that knows about where all the Whatevers are. Your little Whatever Locator site should only ever _pull_ from them. Need a new Whatever in the database? Get BigCentralThing to add it, then pull it in.
As an example, I run https://bettybeta.com/, a select guide to bouldering in Fontainebleau tailored for Women and short folk. We have a list of boulder problems in our database, but no way for anybody (even an admin) to add a new one. Every boulder problem record includes a link to its page at https://bleau.info/, who are dedicated to cataloging every climb in the forest.
Bleau.info is the Single Point of Truth. If one of our users does a First Ascent and wants to log it with us, we'll point them at that other site with instructions on how to get their climb in there, and tell them how to get us to import it. Any other path would lead to our database and the "official" database eventually diverging.
There are half a dozen other websites doing similar things to us. Some do what we do, but the ones that don't quickly find themselves out of sync and end up with a lot more work on their hands.
It's tough because, as the author says, writing the flow to import at our end is really easy (I've written it, before coming to this realization and hiding it again). But doing it is a trap.
This can sometimes be a challenge with OSM, if the way you want to structure data doesn’t align with the way the community has decided to structure data.
I was looking into building a canoeing map based on OSM data. An important thing to map is “is a particular stream between two lakes navigable by canoe?” OSM has a defined canoe=yes/no tag, but it’s supposed to be used for legal access, not navigability.
In the place I wanted to map, canoeing is basally legal everywhere, but there are many streams that no one ever canoes down, because it would be an enormous pain. There isn’t a way to represent this is OSM today, and community tends to be resistant to adding tags that map “subjective” data.
The solution would probably be for me to keep my own database of navigable streams with references to the OSM features, except that OSM features don’t really have stable ids. If another mapper comes along and decides to map an existing stream in more detail, they might choose to delete an existing stream feature and replace it, split it up into multiple features, or join several existing features together — all of which will result in id changes.
What makes a stream navigable and what hinders it? Is it water depth? Is it width? Can those be mapped? Are there edge cases that some people might be able to navigate those waters and others not? Or during certain times?
It sounds like the real issue is the "stable ids" one. You need a Single Point of Truth that matches up stretches of waterway with an ID that won't change. You can then sprinkle .isNavigable and such onto your record along with a reference to that magic ID. But yeah, if they don't provide is, they're not a very good point of truth.
I have to deal with this in my little bouldering database as well, since the source I mentioned does in fact change or reuse its IDs every so often. There's already a periodic update that has to happen on any given boulder problem, to pull down changes from the other end. That also needs a way to self-heal if a record at the other end splits off into two pieces, and my ID now points to the wrong half of it.
It's all stuff you have to keep on top of, but your idea of keeping your own table with each record pointing back to OSM seems like the only path that avoids madness. Even it it does take some work to keep those references up to date.
OpenStreetMap does not and cannot offer permanent IDs. An entity might be mapped in more detail, resulting in something being split in two parts, or change from node to way, with wildly different IDs.
At the same time, a point might be repurposed as something else entirely.
I agree that that's the ideal way to do this, but it's just not always an option.
I run valideparkeren.nl, a crowd sourced overview of all accessible parking spots in the Netherlands. Ideally, there should be a government-owned single source of truth, but the whole reason for starting this is because that source doesn't exist yet. I'm starting to work with the government to eventually hand this project over to them, which was the goal from the start, but the only way to make that happen is by first becoming the source of truth.
Edit: I was also planning to add my data to OSM at some point, but I'm reconsidering that after reading this. I completely understand that they only want high quality data, but in my case I'm convinced that high coverage and slightly lower quality is better than low coverage and high quality. Of course, transparency of how sure we are about the data is vital, so users can decide the risk they want to take.
> I completely understand that they only want high quality data
I worked on "openopeningstijden", where my aim was to improve the data for businesses in OSM and to disclose that in apis and uis.
There were other reasons why I had to pull the plug, but a big difficulty was the understandable conservatism of OSM.
I didn't even want any tagging schemes changed, or the insane dsl for "opening hours" improved, at that time.
All I wanted was to allow obviously "wrong" data to be flagged with a note. Where "wrong" could easily be proven with links to other resources. "The shop is permanently closed. See this url at archive org about their announcement" or "contact details wrong. See their website at url and cross check with BigFoodDeliveryPlatform entry at ..."
Again. Reluctance for such automation is understandable wrt intelectual property rights. It becomes too easy for editors to copy in data that's not allowed.
But the result is that businesses on OSM are poorly represented, often many are missing. That data is hopelessly outdated. And that consumers use proprietary sources and apps to find e.g. a vegetarian restaurant in an unknown city, or to find out if the hairdresser is open at noon.
First of all, I assume you are from the Netherlands (based on the .nl)
The Dutch OSM-community indeed is very conservative, a bit too much IMHO. We Belgians are more relaxed :)
Such a dataset would still be useful, but perhaps in a quality assurance tool, not as map notes that would clog the map. Can you link more information?
Yeah, it's tough because to be the Single Point of Truth, you'd need to be the List of Parking Places in the Netherlands, not just the accessible ones that you care about. You might even need to be The List for Europe.
Being The List of anything is a big, thankless job filled with nonstop toil to keep it up to date. It's not something you want to do unless it's truly your passion or somebody is paying you (a lot) to do it.
The problem is that building to tools to Be The List is pretty straightforward and fun. You only learn about the toil you've signed yourself up after the fact, once people start relying on it.
Curious. You’re curating from the single point of truth (spot). You direct new additions for your site over to spot. How does a submission get into your subset?
Does the submitter add to spot and then shoot you a message to pull by some unique id (author, submitter)? If you’re curating from their database, then spot doesn’t have the schema features to resolve to your subset by filtering alone, right?
Pretend the SPOT is youtube, for an example. Upload your video there, then drop a link to (any variation of) the url into this text field. A quick import script runs, and now we have the video in our database too. Or at least, a record with a link to it and whatever info we need to pull from it, along with extra fields we need to do whatever our website does.
It still lives in both places, but (assuming other websites also use the same Single Point Of Truth) if the title changes or whatever, everything stays synced up everywhere.
In my concrete example, bleau.info doesn't need to know that a given line has never had an ascent by anybody shorter than 165cm. So I write that down on my copy. But I don't need to track anything else about that line to use it and to link to Boolder, 27 Crags and 8a.nu's pages about it.
I haven't seen this in practice. For my site, roughly 100% of the new additions to our database come from people climbing a route that's in the Big Database that we just don't know about yet.
It's just a matter of dropping a link to that route's page into an importer to scrape it in and have it be usable at our end.
But yeah, for a space where the Big Database didn't often have a record for the things your users want to add, I could see it being more of an issue.
This is partially because they import into their own system, and then export->import into OSM. At that point, they're considered an integration which imports data.
If their apps instead submitted directly to OSM (letting submission trickle back down), then the flow would fit into "user submission". Book Corners simply becomes an app though which the user performed the submission, rather than an intermediary exporting data.
Okay, that’s slightly more reasonable it means if I’m going to be rolling stuff back into OSM in a new country I should at least have a quick chat and ensure what I’m doing abides by that country’s regulations etc like South Korea has some notorious law around data residency
> South Korea has some notorious law around data residency
if you're talking about the export restrictions of map data, that only applies to official, government-managed geodata. OSM explicitly doesn't rely on any other maps, so the law doesn't apply.
No, and this is the issue I have with his write up. Yes all points listed are correct. But do they apply in the same rigor and depth for all imports, likely no.
There is a difference between a tiger import or local government address data for thousands of buildings VS a few thousand street libraries. The community review process should/would consider this.
I would also guess that it took some time to investigate all of the requirements and write it down. Might as well have written an import plan and discussed it :-)
That's not how I would summarize it. OSM is only giving them two options: apply to be treated as an automated sync source, or don't submit at all. They're willing to put in work and care but not to handle all the other complex barriers.
>A project like OSM would be bombarded with spam and junk submissions if it didn’t have these barriers to submission.
I'm not sure about this. The barriers to submissions are honestly more likely to cause object and amenity level data to become out of date. Google is able to have such an up to date place database because it relies on a large quantity of crowdsourced submissions, and employs consensus to figure out the truth.
Google employs _engineers_ to figure out the truth. If an import pipeline is causing problems, a higher-up can just say “stop doing that or we’ll stop your pay cheque”.
OSM doesn’t have that option. If it didn’t have these guardrails in place, a million CS students will see some open-looking data (or not even that, maybe something scraped) and throw it into OSM with a cursory Python script, resulting in an almighty mess. We need the process because we don’t have Google’s leverage of “we pay you”.
> Google is able to have such an up to date place database because it relies on a large quantity of crowdsourced submissions
Google denied our address existed[0] for years (which caused several problems with online shops that used Google Maps as the source of truth for address resolution) - took a good few tries to get them to actually add it.
Currently Google Maps is showing an "ALDI charging station" directly across the road when it is, in fact, about a mile down the road.
It's also showing a Chipotle just around the corner when, you'll not be surprised, there is no Chipotle there and, hilariously, the photo it has is of the Chipotle at the O2 Dome a good 10km away.
Absolute clusterfuck of nonsense is Google Maps.
[0] A 1960s London estate which was adequately represented on Google Maps and Streetview, mind.
I work in a warehouse, and if you type our address into Google Maps, it puts the maker right on our building. But they also label the building as a pub, two buildings down the street.
I've submitted corrections, and had a reply saying they were accepted. You can look on street view and the true location of both pub and warehouse are clearly visible, so there'd be no problem verifying my correction. Their map view includes the outline of the pub, as does their satellite view. The pub's been there 10 years, the warehouse since the 1990s. The Street View car has driven through the pub's car park three times.
Having out of date public book boxes isn't really that big of a deal, at least compared to the massive amounts of broken entries, duplicates, and incorrectly licensed data that would end up from free and unrestricted bulk imports.
As it is today you can largely expect OSM POI data to be incomplete or out of date unless you have someone particularly keen on keeping your local area up to date. OSM is best used as a background map layer for your own custom applications, like embedded maps in apps showing your own data overlayed. Rather than an alternative to the google maps app.
I wouldn't use gmap as a benchmark for up to date POIs, at least in some European countries. And I'm not talking about inaccurate opening times; but about places that closed during Covid but are still on the map.
These restrictions are unique to projects doing automated submissions of mass data.
If you're an individual just trying to make the world a better place, you can make an OSM account, go to the website, click, add a thing, hit submit, done. Someone will review it.
Apps like CoMaps and Organic Maps make it super easy to contribute data for businesses, landmarks, and such.
> you can make an OSM account, go to the website, click, add a thing, hit submit, done. Someone will review it.
I will share an unpopular opinion: thil will only make a freeworkforce for those projects who profit from using OSM. And the OSM has surprisingly unreliable data quality for walking around even in the popular tourist places like Tokyo or Kyoto, or New York. I've tried, and it's so worse than GMaps (or Bing) to being completely unusable. No work hours of Starbucks, no local dining cafe, no menu contributions.
It's great for the trail and hike outdiors, sure. But that's very different from the city map, as the nature featutes rarely change, not like city POIs.
This is kinda what StreetComplete is supposed to help with, right? it asks me questions about opening and closing hours of cafes and things.
I would assume that tourist places don't have many locals going by and updating them and tourists don't want to fill in info on that kind of app, but that's mostly an adoption question. If more people contributed more info would be mapped.
Yeah, StreetComplete helps both with adding the data in the first place and then keeping is current, since it will also ask those questions again after a while (which also gives an opportunity to check whether a feature still exists). What's less visible there is whether a cafe exists at all – it's not as good to add missing objects as it is for filling in details of existing objects, at least for the casual user.
It is a very good point though, Honestly we should have tried to solve that. It is badly needed in a map nowdays. At the moment there is no way to map or display these things and I think there should be.
How so? They have a way of storing that specific information, so why shouldn't we (in part) judge it by the completeness of the information they allow people to store?
Hello folks, author of Book Corners here. I just made the whole libraries db available for download (it's OBDL licensed).
It's available from every users dashboard (it's a ~2 MB compressed GeoJSON file). It includes both libraries orignally imported from OSM and additional libraries added by users.
While I still think the whole OSM process to submit back is a bit too much for the time I have available (but I still understand and respect), at least I want to give users an additional option to retrieve the data (in addition to the public API).
If you have any other advices, I will be glad to hear them and, compatibly with my spare time, I will try to implement them (if they make sense and are useful for users) :)
This one sends data straight to OSM. I don't have my own database to "moderate", pictures are saved into a Panoramax-instance.
As users authenticate by logging in with OSM, the boring, risky legal stuff is handled by OpenStreetMap. Uploading images has a small print that says that their images will be reshared under CC-BY-SA.
The part where I suck is in creating a nice, polished website with extra pages, SEO, ...
> There are also important licensing questions. A user’s permission to send a library to OSM is not automatically the same as having a sufficiently clear right to release that factual information under terms compatible with OSM
which would imply that you could not make available your database available for download under ODbL. Yet you have made it available under that license.
Maybe that part of the article needs further editing?
Also, if the data of your tool is now available under ODbL, then the tasks related to importing it into OSM can be effectively performed by anyone, not necessarily you.
> Maybe that part of the article needs further editing?
Yes, I should edit it. I had not though about the bulk download until another user on Mastodon suggested it to me.
> Also, if the data of your tool is now available under ODbL, then the tasks related to importing it into OSM can be effectively performed by anyone, not necessarily you.
Definitely. I would be more than glad if libraries which are not on OSM were added back, I just don't have the required spare time to go through all the process and also maintain it.
Have you considered making OSM itself the submission mechanism?
This could be as simple as having a static guide on how to add bookcases to OSM. Your app could also act as an OSM editor specifically for bookcases (not a general one).
The only difficulty is data than can not be easily represented in key=value pairs, such as timetables for buses. But even this can be resolved by adding external references.
This is trivial to navigate.
Use the notes API to post the likely presence of a public bookcase with details (link to image, description, etc); and the community will slowly ingest the contributions.
Or set up a maproulette challenge with much the same detail.
Or just a geojson dump under an ODBL licence and share with with relevant communities. The third one or something very like it is required if you have mixed ODBL with non ODBL content.
This makes sense, but I wish there was a generally accepted way to share geospatial data and POI data against OSM IDs. e.g. I want to be able to add overlays for noise level, etc. and I find the whole thing too fragmented. The OSM way seems to be the right way, but no one seems to do it. You just maintain your own DB with an optional OSM POI entity identifier etc.
I suppose if you want something badly enough you have to do it yourself, but it's not that the thing doesn't exist though it doesn't seem to. It's that I want others to want it too :)
Can you simply write overlays as GOL files and load them from non-OSM URLs? Assuming you use the appropriate coordinate grid you won’t need POIs to layer the relevant data, and if you choose to annotate OSM’s POIs further, you can save those annotations as a personal GOL that you optionally share with others, and that also enables cross-referencing derivations if you use (unverified: trust isn’t the point, uniqueness is) signing key fingerprint as the GOL author GUID.
I think the piece I’m confused most about is why the little free library website’s map isn’t the place to log little free libraries.
That being said, it makes sense that OSM has guidelines like this in place to ensure the data quality stays, well, quality.
I feel like having such a map at all is kind of antithetical to building community. Shows you don't care enough to take a stroll in your 'hood to find such a one by happenstance (or to find neighbors to populate yours) and are hunting them down to share your self-published slop or something. Besides, it makes things a hell of a lot easier for griefers and trolls.
I've always been confused why OSM doesn't design a cheap open source stereo imager, some finetuned modern YOLO neural network could detect and blur the glass parts of windows, doorbells, nameplates and humans in the scene.
It could use GPS for rough geolocation, and 3D models of all the scenery could be generated. New contribution traces would contain changes compared to the past. Volunteers could request new paths it would like to see explored, and OSM could propose paths that cut through or ride along segments of recently submitted recordings of other users, to check if those changes are real, without checking the whole suspected recording.
Separate ground truth recording from its interpretation into mappable concepts, going out to make a recording or observation is a different task from deciding how to canonicalize the content.
Are you describing what Mapillary is doing ? It's a crowdsourced street level imagery provider (owned by Facebook since 2020, but still contributing to OSM), but they do auto-detection of feature (you can see that in the online OSM editor: Map Data > Photo Overlays > Mapillary > Map Features).
I vaguely remember a blog post about doing photogrammetry out of their images, but I'm not entirely sure about it.
the compute for rigidly separated binocular vision is much much more compute efficient and precise than having to guesstimate positions from monocular + IMU.
Well, first step here is to demonstrate that you can do that neural network part. I think that would be a really valuable tool for more projects than openstreetmaps - google might even want to buy it.
OpenStreetMap is a 2D map with limited 3D features, not a streetview alternative. For crowdsourced street view there are services like Mapillary (now owned by Meta) and Panoramax, both of which are cleared for use in mapping in OSM (and in case of Panoramax which is self-hostable, often run by a local OSM community).
I appreciate OSM for maintaining a higher data quality bar than other projects (Overture places are mostly junk outside of USA), but it's also just artificially limiting itself by not allowing streamlined paths to data contributions.
Like I have a list of a few hundred osm places websites that are clearly scam sites. I should be going one by one and filing them manually but I found this via a spam filter and it’s very robust. I should have a way of getting these scam sites reported to osm.
You could share it here and let more people remove the spam sites. If everybody removes a couple in no-time all will be removed.
Which is probably what might happen if you would have the option to "report them to osm". Someone should check things, because maybe you are a new person to them, maybe the spam filter might not be as robust as you might think, etc. Even if you are perfect, other submission might not be, so someone needs to check/review/etc.
It is streamlined for individuals, and I have no interest in it being streamlined for anything else.
I'd rather no data than bad data. There is no seperate baby & bath water. If a bulk source of data contains an unknown mix of good and bad data, that is all one big single item of bad data that is of no use to anyone.
I’m grateful that a principled group of people run OSM. Much like I’m grateful that a principled group run Wikipedia. But the rigidness has costs that I don’t think are being appreciated.
Is there a path forward to writing a guide for how users can contribute back to OSM if they wish? A "thank you for your contribution, if you wish to learn more about OSM and how our maps are built click here" => go to a blog post which describes the relationship with OSM, why Book Corners can't be liable for contributing data back to OSM, and how a user could do that if they wanted.
Maybe an alternate process could use streetcomplete? Make the imported data available in streetcomplete for validation, then include it only after x validations?
Mapy.com does this and I often see and resolve these notes on streetcomplete. It's nice setup, they provide a suggestion and location, it takes little effort to check if you are walking by and just reply to the note. Some enthusiastic editor (can be you back home, but it's annoying to do on a phone screen) then can make the edit and resolve the note.
Yes, the data source is clearly useful, so not having it is a loss. But an edit is large enough a privilege to require all these checks.
Perhaps it's not that OSM that needs a signal, but perhaps it's that there's an opportunity for an open map that has many signals, of which OSM and Book Corners are examples.
Laying your own datasets on top of OSM is probably the most common use case. OSM isn't much of a replacement to Google and Apple maps as it is a way to build your own custom maps.
This can probably be added to the toolchain pretty easily when non-OSM "signals" are converted to OSM format first. All you need then is an OSM data merging tool, all other tools will then get the same data format as before.
The article is definitely write or at least rewritten or heavily editorialised by an LLM. You can confirm this by skimming the author’s previous articles and compare it to this one, specially pre-2021.
Come on, the LLM language is all over the article:
> The workflow I had in mind was deliberately cautious
> The code was not the difficult part.
> These requirements are not a one-time form to complete and forget. They create an ongoing responsibility around the account, the documented process, community feedback, failures, and potential reversions.
Just a few examples. And yes, Pangram 4 also flags it as 100% LLM written. I don't mind being downvoted or flagged, but I think more people should be aware of the LLM style, even if they're ok with it being used without any disclaimer. It's honestly sad that nowadays people on HN cannot recognize this style.
Perhaps you haven't used LLMs enough, this style is extremely common with Claude. And can you link me a Pangram 4 example where it has lots of false positives? The main reason why Pangram is supposed to be reliable in the first place is that they trade false positives for false negatives.
Also I don't get the point why you would want them easily scrapeable to begin with.
If you have eyes to read and can walk across YOUR neighborhoods, you would have eventually seen one. I thought that was the whole point? They are not exactly hidden.
I find those public bookcase incidental low barrier offerings to a reading exprience you just stumble upon, nothing you would seek out on a map.
Why would you want to have more people from anywhere else possibly raid them for anything useful?
This way nobody is inclined to put anything worthwhile in there if it just ends up outside of their communities. Not that I'm in general pessimistic, just would like to know what is gained by putting the exact location into a map and not just have a list of neigborhoods with a book corner and optional photo on that neigborhood level, if people want to network around them.
Those people traveling to them are probably just as likely to put something in as a more local passer by. And I'm not sure why I would care that someone the next suburb over read my book. As long as someone is enjoying it, that's the whole point of donating.
Recently it's seemed like (very roughly) half the Little Free Libraries I see aren't in OSM.
I sometimes edit OSM, so I'm wondering whether any public bookshelf omissions are intentional, and adding the shelves would be unwelcome by the people running them.
I see OSM as a mapping application. It is in the real world, publicly visible for a long period of time, and there is no law preventing sharing the information (like military stuff) it can be in the mapping application (decide what is relevant enough is another story).
I would not include the wishes of the people into it, because there are various groups that might have strange ideas (dunno, like only churches of some type should be shown; or no shops of a certain kind because it hurts my feelings, etc.).
I personally would not like any edits that advances the wishes of "any" group of people (like removing features). I do not like all things that exists on maps, but I will not remove them because of my feelings.
The few places I have seen book nooks like this, they end filled up with religious literature which is usually not the intent of the person who started it.
I run a website that's in a similar situation, and it sounds like the author is close to the correct solution, but needs to take things one step further:
There needs to be a Single Point of Truth, and it should not be your little website.
BigCentralThing should be the official database that knows about where all the Whatevers are. Your little Whatever Locator site should only ever _pull_ from them. Need a new Whatever in the database? Get BigCentralThing to add it, then pull it in.
As an example, I run https://bettybeta.com/, a select guide to bouldering in Fontainebleau tailored for Women and short folk. We have a list of boulder problems in our database, but no way for anybody (even an admin) to add a new one. Every boulder problem record includes a link to its page at https://bleau.info/, who are dedicated to cataloging every climb in the forest.
Bleau.info is the Single Point of Truth. If one of our users does a First Ascent and wants to log it with us, we'll point them at that other site with instructions on how to get their climb in there, and tell them how to get us to import it. Any other path would lead to our database and the "official" database eventually diverging.
There are half a dozen other websites doing similar things to us. Some do what we do, but the ones that don't quickly find themselves out of sync and end up with a lot more work on their hands.
It's tough because, as the author says, writing the flow to import at our end is really easy (I've written it, before coming to this realization and hiding it again). But doing it is a trap.
This can sometimes be a challenge with OSM, if the way you want to structure data doesn’t align with the way the community has decided to structure data.
I was looking into building a canoeing map based on OSM data. An important thing to map is “is a particular stream between two lakes navigable by canoe?” OSM has a defined canoe=yes/no tag, but it’s supposed to be used for legal access, not navigability.
In the place I wanted to map, canoeing is basally legal everywhere, but there are many streams that no one ever canoes down, because it would be an enormous pain. There isn’t a way to represent this is OSM today, and community tends to be resistant to adding tags that map “subjective” data.
The solution would probably be for me to keep my own database of navigable streams with references to the OSM features, except that OSM features don’t really have stable ids. If another mapper comes along and decides to map an existing stream in more detail, they might choose to delete an existing stream feature and replace it, split it up into multiple features, or join several existing features together — all of which will result in id changes.
> there are many streams that no one ever canoes down, because it would be an enormous pain. There isn’t a way to represent this is OSM today
Had you considered `canoe=discouraged`?
https://wiki.openstreetmap.org/wiki/Key:canoe#Access_values
> paddling is technically allowed but may often be unsafe or impractical (e.g. open water or shallow creeks)
What makes a stream navigable and what hinders it? Is it water depth? Is it width? Can those be mapped? Are there edge cases that some people might be able to navigate those waters and others not? Or during certain times?
If you work this out, you'll get there
It sounds like the real issue is the "stable ids" one. You need a Single Point of Truth that matches up stretches of waterway with an ID that won't change. You can then sprinkle .isNavigable and such onto your record along with a reference to that magic ID. But yeah, if they don't provide is, they're not a very good point of truth.
I have to deal with this in my little bouldering database as well, since the source I mentioned does in fact change or reuse its IDs every so often. There's already a periodic update that has to happen on any given boulder problem, to pull down changes from the other end. That also needs a way to self-heal if a record at the other end splits off into two pieces, and my ID now points to the wrong half of it.
It's all stuff you have to keep on top of, but your idea of keeping your own table with each record pointing back to OSM seems like the only path that avoids madness. Even it it does take some work to keep those references up to date.
OpenStreetMap does not and cannot offer permanent IDs. An entity might be mapped in more detail, resulting in something being split in two parts, or change from node to way, with wildly different IDs.
At the same time, a point might be repurposed as something else entirely.
You could store the ID in a tag. They already do this for e.g. wikidata[1] and gnis[2].
Though I kinda doubt they want to support maintaining these for every single project that wants one.
[1]: https://wiki.openstreetmap.org/wiki/Key:wikidata
[2]: https://wiki.openstreetmap.org/wiki/Key:gnis:feature_id
I agree that that's the ideal way to do this, but it's just not always an option.
I run valideparkeren.nl, a crowd sourced overview of all accessible parking spots in the Netherlands. Ideally, there should be a government-owned single source of truth, but the whole reason for starting this is because that source doesn't exist yet. I'm starting to work with the government to eventually hand this project over to them, which was the goal from the start, but the only way to make that happen is by first becoming the source of truth.
Edit: I was also planning to add my data to OSM at some point, but I'm reconsidering that after reading this. I completely understand that they only want high quality data, but in my case I'm convinced that high coverage and slightly lower quality is better than low coverage and high quality. Of course, transparency of how sure we are about the data is vital, so users can decide the risk they want to take.
> I completely understand that they only want high quality data
I worked on "openopeningstijden", where my aim was to improve the data for businesses in OSM and to disclose that in apis and uis.
There were other reasons why I had to pull the plug, but a big difficulty was the understandable conservatism of OSM. I didn't even want any tagging schemes changed, or the insane dsl for "opening hours" improved, at that time.
All I wanted was to allow obviously "wrong" data to be flagged with a note. Where "wrong" could easily be proven with links to other resources. "The shop is permanently closed. See this url at archive org about their announcement" or "contact details wrong. See their website at url and cross check with BigFoodDeliveryPlatform entry at ..."
Again. Reluctance for such automation is understandable wrt intelectual property rights. It becomes too easy for editors to copy in data that's not allowed.
But the result is that businesses on OSM are poorly represented, often many are missing. That data is hopelessly outdated. And that consumers use proprietary sources and apps to find e.g. a vegetarian restaurant in an unknown city, or to find out if the hairdresser is open at noon.
First of all, I assume you are from the Netherlands (based on the .nl)
The Dutch OSM-community indeed is very conservative, a bit too much IMHO. We Belgians are more relaxed :)
Such a dataset would still be useful, but perhaps in a quality assurance tool, not as map notes that would clog the map. Can you link more information?
Yeah, it's tough because to be the Single Point of Truth, you'd need to be the List of Parking Places in the Netherlands, not just the accessible ones that you care about. You might even need to be The List for Europe.
Being The List of anything is a big, thankless job filled with nonstop toil to keep it up to date. It's not something you want to do unless it's truly your passion or somebody is paying you (a lot) to do it.
The problem is that building to tools to Be The List is pretty straightforward and fun. You only learn about the toil you've signed yourself up after the fact, once people start relying on it.
Good luck!
Curious. You’re curating from the single point of truth (spot). You direct new additions for your site over to spot. How does a submission get into your subset?
Does the submitter add to spot and then shoot you a message to pull by some unique id (author, submitter)? If you’re curating from their database, then spot doesn’t have the schema features to resolve to your subset by filtering alone, right?
Pretend the SPOT is youtube, for an example. Upload your video there, then drop a link to (any variation of) the url into this text field. A quick import script runs, and now we have the video in our database too. Or at least, a record with a link to it and whatever info we need to pull from it, along with extra fields we need to do whatever our website does.
It still lives in both places, but (assuming other websites also use the same Single Point Of Truth) if the title changes or whatever, everything stays synced up everywhere.
In my concrete example, bleau.info doesn't need to know that a given line has never had an ascent by anybody shorter than 165cm. So I write that down on my copy. But I don't need to track anything else about that line to use it and to link to Boolder, 27 Crags and 8a.nu's pages about it.
> Get BigCentralThing to add it
Making a workflow central to your business dependent on the goodwill and efficiency of an external entity is a very bad idea!
It's good that you've found a partner who is willing to put in the work, but that is luck rather than what normally happens.
This way you get less contributions.
I haven't seen this in practice. For my site, roughly 100% of the new additions to our database come from people climbing a route that's in the Big Database that we just don't know about yet.
It's just a matter of dropping a link to that route's page into an importer to scrape it in and have it be usable at our end.
But yeah, for a space where the Big Database didn't often have a record for the things your users want to add, I could see it being more of an issue.
In summary: Because OSM requires work and care to be put into the data submission plan, which isn’t worth it.
A project like OSM would be bombarded with spam and junk submissions if it didn’t have these barriers to submission. Understandable.
This is partially because they import into their own system, and then export->import into OSM. At that point, they're considered an integration which imports data.
If their apps instead submitted directly to OSM (letting submission trickle back down), then the flow would fit into "user submission". Book Corners simply becomes an app though which the user performed the submission, rather than an intermediary exporting data.
Such as what I make: https://mapcomplete.org/bookcases.html?
It's not really considered an import if a person is looking at all of the changes. They would just need a clear license from their users.
(an import is when there isn't complete review...)
I get this perspective. But I also find the standards to be onerous. They were honestly fine until this one:
> Contact the relevant local communities affected by the contributions
Do I literally have to do new local community outreach every little town one of my bookcases shows up in? Ridiculous if so
Refers to OSM communities, not the literal physical local communities offline.
If every contribution required community outreach, we wouldn’t see any contributions.
Generally this means “national” and the sub-board on the OSM discussion forum.
Okay, that’s slightly more reasonable it means if I’m going to be rolling stuff back into OSM in a new country I should at least have a quick chat and ensure what I’m doing abides by that country’s regulations etc like South Korea has some notorious law around data residency
> South Korea has some notorious law around data residency
if you're talking about the export restrictions of map data, that only applies to official, government-managed geodata. OSM explicitly doesn't rely on any other maps, so the law doesn't apply.
No, and this is the issue I have with his write up. Yes all points listed are correct. But do they apply in the same rigor and depth for all imports, likely no. There is a difference between a tiger import or local government address data for thousands of buildings VS a few thousand street libraries. The community review process should/would consider this. I would also guess that it took some time to investigate all of the requirements and write it down. Might as well have written an import plan and discussed it :-)
That's not how I would summarize it. OSM is only giving them two options: apply to be treated as an automated sync source, or don't submit at all. They're willing to put in work and care but not to handle all the other complex barriers.
There's another path that the author hasn't considered, which is making users contribute directly to OSM.
To contribute a bookcase, ask them to create an OSM account, log in with OAuth, and send the edit straight to OSM.
Book corners becomes just a custom view and editor over OSM data, instead of being its own database.
Users are now responsible for the data they submit (OSM already has tools to deal with vandalism etc...).
Throwing out half the assistance the site is able to give and giving up control over the process is... technically an option...
>A project like OSM would be bombarded with spam and junk submissions if it didn’t have these barriers to submission.
I'm not sure about this. The barriers to submissions are honestly more likely to cause object and amenity level data to become out of date. Google is able to have such an up to date place database because it relies on a large quantity of crowdsourced submissions, and employs consensus to figure out the truth.
Google employs _engineers_ to figure out the truth. If an import pipeline is causing problems, a higher-up can just say “stop doing that or we’ll stop your pay cheque”.
OSM doesn’t have that option. If it didn’t have these guardrails in place, a million CS students will see some open-looking data (or not even that, maybe something scraped) and throw it into OSM with a cursory Python script, resulting in an almighty mess. We need the process because we don’t have Google’s leverage of “we pay you”.
> Google employs _engineers_ to figure out the truth.
and people say hacker news has no sense of humour
Another approach is to simply accept low quality data into the database, yet have some kind of filtering during the viewing stage.
Eg. Draw me a map, but include only data points tagged with 'osm-license-verified-and-spam-filtered'.
That way users of the data get to decide their own tradeoff between legal risks, data freshness, spam, etc.
> Google is able to have such an up to date place database because it relies on a large quantity of crowdsourced submissions
Google denied our address existed[0] for years (which caused several problems with online shops that used Google Maps as the source of truth for address resolution) - took a good few tries to get them to actually add it.
Currently Google Maps is showing an "ALDI charging station" directly across the road when it is, in fact, about a mile down the road.
It's also showing a Chipotle just around the corner when, you'll not be surprised, there is no Chipotle there and, hilariously, the photo it has is of the Chipotle at the O2 Dome a good 10km away.
Absolute clusterfuck of nonsense is Google Maps.
[0] A 1960s London estate which was adequately represented on Google Maps and Streetview, mind.
It's bizarre, isn't it?
I work in a warehouse, and if you type our address into Google Maps, it puts the maker right on our building. But they also label the building as a pub, two buildings down the street.
I've submitted corrections, and had a reply saying they were accepted. You can look on street view and the true location of both pub and warehouse are clearly visible, so there'd be no problem verifying my correction. Their map view includes the outline of the pub, as does their satellite view. The pub's been there 10 years, the warehouse since the 1990s. The Street View car has driven through the pub's car park three times.
And yet, the pub marker stays in the wrong place.
Having out of date public book boxes isn't really that big of a deal, at least compared to the massive amounts of broken entries, duplicates, and incorrectly licensed data that would end up from free and unrestricted bulk imports.
As it is today you can largely expect OSM POI data to be incomplete or out of date unless you have someone particularly keen on keeping your local area up to date. OSM is best used as a background map layer for your own custom applications, like embedded maps in apps showing your own data overlayed. Rather than an alternative to the google maps app.
I wouldn't use gmap as a benchmark for up to date POIs, at least in some European countries. And I'm not talking about inaccurate opening times; but about places that closed during Covid but are still on the map.
To be clear to folks not familiar with OSM:
These restrictions are unique to projects doing automated submissions of mass data.
If you're an individual just trying to make the world a better place, you can make an OSM account, go to the website, click, add a thing, hit submit, done. Someone will review it.
Apps like CoMaps and Organic Maps make it super easy to contribute data for businesses, landmarks, and such.
> you can make an OSM account, go to the website, click, add a thing, hit submit, done. Someone will review it.
I will share an unpopular opinion: thil will only make a freeworkforce for those projects who profit from using OSM. And the OSM has surprisingly unreliable data quality for walking around even in the popular tourist places like Tokyo or Kyoto, or New York. I've tried, and it's so worse than GMaps (or Bing) to being completely unusable. No work hours of Starbucks, no local dining cafe, no menu contributions.
It's great for the trail and hike outdiors, sure. But that's very different from the city map, as the nature featutes rarely change, not like city POIs.
One of the nice things about OSM is you can profit from it from it if you want to.
I don’t think you’ll find even most the die hard OSM fan disagree with you on the POI data.
On the other hand, the road network in OSM is very high quality. As you point out, it’s used and contributed to by companies like Lyft and Amazon.
The fact that anyone can download an accurate detailed global road network for free is pretty crazy.
This is kinda what StreetComplete is supposed to help with, right? it asks me questions about opening and closing hours of cafes and things.
I would assume that tourist places don't have many locals going by and updating them and tourists don't want to fill in info on that kind of app, but that's mostly an adoption question. If more people contributed more info would be mapped.
Yeah, StreetComplete helps both with adding the data in the first place and then keeping is current, since it will also ask those questions again after a while (which also gives an opportunity to check whether a feature still exists). What's less visible there is whether a cafe exists at all – it's not as good to add missing objects as it is for filling in details of existing objects, at least for the casual user.
It's fine to make a project that will only be interesting if people choose to contribute, and it's fine if people want to contribute.
> No work hours of Starbucks, no local dining cafe, no menu contributions.
This moves the goal posts of OSM
Well, those things can be added:
But they can easily become outdated again. Still, local QA via StreetComplete, MapComplete, EveryDoor and a multitude of other apps is always needed.It is a very good point though, Honestly we should have tried to solve that. It is badly needed in a map nowdays. At the moment there is no way to map or display these things and I think there should be.
How so? They have a way of storing that specific information, so why shouldn't we (in part) judge it by the completeness of the information they allow people to store?
Hello folks, author of Book Corners here. I just made the whole libraries db available for download (it's OBDL licensed).
It's available from every users dashboard (it's a ~2 MB compressed GeoJSON file). It includes both libraries orignally imported from OSM and additional libraries added by users.
While I still think the whole OSM process to submit back is a bit too much for the time I have available (but I still understand and respect), at least I want to give users an additional option to retrieve the data (in addition to the public API).
If you have any other advices, I will be glad to hear them and, compatibly with my spare time, I will try to implement them (if they make sense and are useful for users) :)
Thanks
How would you feel about integrating https://mapcomplete.org/bookcases.html? (which I make) as means of contribution?
This one sends data straight to OSM. I don't have my own database to "moderate", pictures are saved into a Panoramax-instance.
As users authenticate by logging in with OSM, the boring, risky legal stuff is handled by OpenStreetMap. Uploading images has a small print that says that their images will be reshared under CC-BY-SA.
The part where I suck is in creating a nice, polished website with extra pages, SEO, ...
In the article, you claim:
> There are also important licensing questions. A user’s permission to send a library to OSM is not automatically the same as having a sufficiently clear right to release that factual information under terms compatible with OSM
which would imply that you could not make available your database available for download under ODbL. Yet you have made it available under that license.
Maybe that part of the article needs further editing?
Also, if the data of your tool is now available under ODbL, then the tasks related to importing it into OSM can be effectively performed by anyone, not necessarily you.
> Maybe that part of the article needs further editing?
Yes, I should edit it. I had not though about the bulk download until another user on Mastodon suggested it to me.
> Also, if the data of your tool is now available under ODbL, then the tasks related to importing it into OSM can be effectively performed by anyone, not necessarily you.
Definitely. I would be more than glad if libraries which are not on OSM were added back, I just don't have the required spare time to go through all the process and also maintain it.
Have you considered making OSM itself the submission mechanism? This could be as simple as having a static guide on how to add bookcases to OSM. Your app could also act as an OSM editor specifically for bookcases (not a general one).
I haven't. That would be a completely different app which I'm not interested in working on, I'm afraid.
Presumably they hope to provide richer data (i.e. photograph of the library) that OSM isn't interested in.
OSM is interested in pictures, even of bookcases:
https://wiki.openstreetmap.org/wiki/Key:image
The only difficulty is data than can not be easily represented in key=value pairs, such as timetables for buses. But even this can be resolved by adding external references.
This is the correct approach.
This is trivial to navigate. Use the notes API to post the likely presence of a public bookcase with details (link to image, description, etc); and the community will slowly ingest the contributions.
Or set up a maproulette challenge with much the same detail.
Or just a geojson dump under an ODBL licence and share with with relevant communities. The third one or something very like it is required if you have mixed ODBL with non ODBL content.
This makes sense, but I wish there was a generally accepted way to share geospatial data and POI data against OSM IDs. e.g. I want to be able to add overlays for noise level, etc. and I find the whole thing too fragmented. The OSM way seems to be the right way, but no one seems to do it. You just maintain your own DB with an optional OSM POI entity identifier etc.
I suppose if you want something badly enough you have to do it yourself, but it's not that the thing doesn't exist though it doesn't seem to. It's that I want others to want it too :)
Can you simply write overlays as GOL files and load them from non-OSM URLs? Assuming you use the appropriate coordinate grid you won’t need POIs to layer the relevant data, and if you choose to annotate OSM’s POIs further, you can save those annotations as a personal GOL that you optionally share with others, and that also enables cross-referencing derivations if you use (unverified: trust isn’t the point, uniqueness is) signing key fingerprint as the GOL author GUID.
I think the piece I’m confused most about is why the little free library website’s map isn’t the place to log little free libraries. That being said, it makes sense that OSM has guidelines like this in place to ensure the data quality stays, well, quality.
Because you need to pay "Little Free Library Corp." to become an official LFL and to get onto that map: https://littlefreelibrary.org/stewards/registration/
Getting an official "charter sign" will cost you 50 bucks.
I feel like having such a map at all is kind of antithetical to building community. Shows you don't care enough to take a stroll in your 'hood to find such a one by happenstance (or to find neighbors to populate yours) and are hunting them down to share your self-published slop or something. Besides, it makes things a hell of a lot easier for griefers and trolls.
I've always been confused why OSM doesn't design a cheap open source stereo imager, some finetuned modern YOLO neural network could detect and blur the glass parts of windows, doorbells, nameplates and humans in the scene.
It could use GPS for rough geolocation, and 3D models of all the scenery could be generated. New contribution traces would contain changes compared to the past. Volunteers could request new paths it would like to see explored, and OSM could propose paths that cut through or ride along segments of recently submitted recordings of other users, to check if those changes are real, without checking the whole suspected recording.
Separate ground truth recording from its interpretation into mappable concepts, going out to make a recording or observation is a different task from deciding how to canonicalize the content.
It’s a volunteer org. Sounds like you’re volunteering to build that and pay for the imagery etc. Let us know when you’re done.
Are you describing what Mapillary is doing ? It's a crowdsourced street level imagery provider (owned by Facebook since 2020, but still contributing to OSM), but they do auto-detection of feature (you can see that in the online OSM editor: Map Data > Photo Overlays > Mapillary > Map Features).
I vaguely remember a blog post about doing photogrammetry out of their images, but I'm not entirely sure about it.
I was talking about photogrammetry, preferably with 2 global shutter synchronous image sensors, attached to the ends of say a 1 m pole.
Don't need global shutter just use a bit more math and IMU data with frequency in excess of the frame rate.
Make it work with the recording capabilities of a good modern phone and handle the compute in non-real-time.
the compute for rigidly separated binocular vision is much much more compute efficient and precise than having to guesstimate positions from monocular + IMU.
Well, first step here is to demonstrate that you can do that neural network part. I think that would be a really valuable tool for more projects than openstreetmaps - google might even want to buy it.
OpenStreetMap is a 2D map with limited 3D features, not a streetview alternative. For crowdsourced street view there are services like Mapillary (now owned by Meta) and Panoramax, both of which are cleared for use in mapping in OSM (and in case of Panoramax which is self-hostable, often run by a local OSM community).
Have you suggested this to the folks at Panoramax? https://panoramax.fr/
They're already covering the blurring, as well as detecting useful objects in imagery (traffic signs for a start).
I appreciate OSM for maintaining a higher data quality bar than other projects (Overture places are mostly junk outside of USA), but it's also just artificially limiting itself by not allowing streamlined paths to data contributions.
I try to create such a thing with https://mapcomplete.org ; StreetComplete.app is also beginner friendly (but a different approach)
Yes, you'll always need a user account
The streamlined path has always been to verify your contributions in the field. Everything else is of dubious quality.
It seems like the path is streamlined once they know who you are.
This sort of feels like a generational thing but I would have flown over.
Like I have a list of a few hundred osm places websites that are clearly scam sites. I should be going one by one and filing them manually but I found this via a spam filter and it’s very robust. I should have a way of getting these scam sites reported to osm.
You could share it here and let more people remove the spam sites. If everybody removes a couple in no-time all will be removed.
Which is probably what might happen if you would have the option to "report them to osm". Someone should check things, because maybe you are a new person to them, maybe the spam filter might not be as robust as you might think, etc. Even if you are perfect, other submission might not be, so someone needs to check/review/etc.
It is streamlined for individuals, and I have no interest in it being streamlined for anything else.
I'd rather no data than bad data. There is no seperate baby & bath water. If a bulk source of data contains an unknown mix of good and bad data, that is all one big single item of bad data that is of no use to anyone.
I’m grateful that a principled group of people run OSM. Much like I’m grateful that a principled group run Wikipedia. But the rigidness has costs that I don’t think are being appreciated.
Is there a path forward to writing a guide for how users can contribute back to OSM if they wish? A "thank you for your contribution, if you wish to learn more about OSM and how our maps are built click here" => go to a blog post which describes the relationship with OSM, why Book Corners can't be liable for contributing data back to OSM, and how a user could do that if they wanted.
Maybe an alternate process could use streetcomplete? Make the imported data available in streetcomplete for validation, then include it only after x validations?
OSM needs a way to ingest "signals" instead of edits.
OSM has the notes feature which sounds like what OP should have used
> Example of a valid and useful note can be "a new road was constructed here" or "this shop is closed and does not exist anymore".
https://wiki.openstreetmap.org/wiki/Notes
Mapy.com does this and I often see and resolve these notes on streetcomplete. It's nice setup, they provide a suggestion and location, it takes little effort to check if you are walking by and just reply to the note. Some enthusiastic editor (can be you back home, but it's annoying to do on a phone screen) then can make the edit and resolve the note.
Yes, the data source is clearly useful, so not having it is a loss. But an edit is large enough a privilege to require all these checks.
Perhaps it's not that OSM that needs a signal, but perhaps it's that there's an opportunity for an open map that has many signals, of which OSM and Book Corners are examples.
Laying your own datasets on top of OSM is probably the most common use case. OSM isn't much of a replacement to Google and Apple maps as it is a way to build your own custom maps.
This can probably be added to the toolchain pretty easily when non-OSM "signals" are converted to OSM format first. All you need then is an OSM data merging tool, all other tools will then get the same data format as before.
You could try pinging an OSM dev to see if they're willing to take over the work laterally before calling it quits
Good product management is mostly about saying No.
[dead]
Fully LLM-written article, maybe that's another reason :)
I highly doubt that. What tells do you see that lends to probability of it being llm generated content?
Is it possible that we (myself included) have become conditioned to think content is llm generated as the default opinion?
The article is definitely write or at least rewritten or heavily editorialised by an LLM. You can confirm this by skimming the author’s previous articles and compare it to this one, specially pre-2021.
Come on, the LLM language is all over the article:
> The workflow I had in mind was deliberately cautious
> The code was not the difficult part.
> These requirements are not a one-time form to complete and forget. They create an ongoing responsibility around the account, the documented process, community feedback, failures, and potential reversions.
Just a few examples. And yes, Pangram 4 also flags it as 100% LLM written. I don't mind being downvoted or flagged, but I think more people should be aware of the LLM style, even if they're ok with it being used without any disclaimer. It's honestly sad that nowadays people on HN cannot recognize this style.
What about your examples has an llm tell?
I don't trust pangram 4 much. There was a post here earlier confirming it fails for many others.
Perhaps you haven't used LLMs enough, this style is extremely common with Claude. And can you link me a Pangram 4 example where it has lots of false positives? The main reason why Pangram is supposed to be reliable in the first place is that they trade false positives for false negatives.
The way the text flows, the section headings that have zero creativity...
[flagged]
Also I don't get the point why you would want them easily scrapeable to begin with. If you have eyes to read and can walk across YOUR neighborhoods, you would have eventually seen one. I thought that was the whole point? They are not exactly hidden. I find those public bookcase incidental low barrier offerings to a reading exprience you just stumble upon, nothing you would seek out on a map.
Why would you want to have more people from anywhere else possibly raid them for anything useful? This way nobody is inclined to put anything worthwhile in there if it just ends up outside of their communities. Not that I'm in general pessimistic, just would like to know what is gained by putting the exact location into a map and not just have a list of neigborhoods with a book corner and optional photo on that neigborhood level, if people want to network around them.
Those people traveling to them are probably just as likely to put something in as a more local passer by. And I'm not sure why I would care that someone the next suburb over read my book. As long as someone is enjoying it, that's the whole point of donating.
Recently it's seemed like (very roughly) half the Little Free Libraries I see aren't in OSM.
I sometimes edit OSM, so I'm wondering whether any public bookshelf omissions are intentional, and adding the shelves would be unwelcome by the people running them.
I see OSM as a mapping application. It is in the real world, publicly visible for a long period of time, and there is no law preventing sharing the information (like military stuff) it can be in the mapping application (decide what is relevant enough is another story).
I would not include the wishes of the people into it, because there are various groups that might have strange ideas (dunno, like only churches of some type should be shown; or no shops of a certain kind because it hurts my feelings, etc.).
I personally would not like any edits that advances the wishes of "any" group of people (like removing features). I do not like all things that exists on maps, but I will not remove them because of my feelings.
The few places I have seen book nooks like this, they end filled up with religious literature which is usually not the intent of the person who started it.