One comment on this: *most of these data providers would be unhappy with KiCad providing an easy way for users to scrape their data, as it could cost them a lot of money*
Actually, Mouser , Digikey, Element14, among a few other already provide this since a long time for free but with a limit on maximum number of calls or records retrieved per day, based on the user's registered account. Though this quota is sufficient say ~100 searches per day. Recently LCSC stopped their free service and only allows customers with a VERY VERY LARGE order amount history. (think CM's), Other retailers require paid subscriptions. These services often involve OAuth2. and that probably is beyond scope for KicAD and even many users as it would need to manage call backs, client secrets, access token, refresh tokens etc...That is a complex affair. Therefor I documented in the proposals that an "intermediate" service running locally or on a remote server implemented by the user (not kicad) to handle the transformation between the call to the API and the service's data and also in there manage authentication (call backs, client credentials, access token and refresh tokens etc..). This part could e provided by external open source projects and out of scope for KiCad. KiCad just makes the API call to that "intermediate" service based on the KicaD API spec. On Thursday, July 13, 2023 at 11:45:39 PM UTC+8 Symdeb wrote: > The word "instead" was probably a bit poorly chosen. Surely, retaining > ODBC makes sense to make since it's already there and people are using that. > No objection on plugins or any variant of solution. As any proposal , the > document was just for feedback, discussion, consideration. Nothing set in > stone. > It was kept as generic as possible w/o restricting the type of > implementation. > > > > > > > > On Thursday, July 13, 2023 at 9:55:02 PM UTC+8 j...@craftyjon.com wrote: > >> There are a lot of different concepts and ideas here being thrown around. >> >> First of all, to reiterate what I said on the linked issue: No REST API >> will be accepted "instead of" ODBC. The existing ODBC system will continue >> to exist as-is to support users for whom it is a better solution than REST. >> >> Second, the scope of any project being developed by someone who hasn't >> worked on KiCad before really should be limited, as it will take time for >> someone to learn the codebase and build a good working relationship with >> the rest of the team. To that end, I do not think it is in scope to >> consider any functionality not already supported by the KiCad library >> system. So, change management / version control are _not_ something I >> would worry about at this point when defining the API interface with KiCad. >> >> If someone wants to build a change management / version control system >> outside of KiCad (as part of whatever web service is exposing this proposed >> REST API) that is fine. >> >> If you or anyone else is serious about pursuing this, I would recommend >> the next step is to continue to build out the API proposal document, based >> on APIs required to build a KiCad symbol library plugin. Once we are in >> agreement on that, a proof-of-concept could be built for testing and review. >> >> -Jon >> >> On Thu, Jul 13, 2023 at 12:46 AM Symdeb <happy.d...@gmail.com> wrote: >> >>> There were several conversations in the past on Kicad Info, Inventree >>> Github and Kicad gitlab. >>> Here is a functional specification proposal for an API instead of ODBC. >>> This built on top the proposed search for parts feature on gitlab , as >>> follows >>> >>> 1. Searching for parts from any resource . The API implementation >>> needs to be provided, not in scope. Specification adds request/response >>> proposal >>> 2. Change (version) management between CAD and remote systems for >>> footprints, symbols, models. Specification adds request/response >>> proposals >>> 3. Part to symbol/footprint/model mapping between remote system and >>> the CAD. Specification adds request/response proposals ad version >>> control >>> logic. >>> >>> "remote system" term is used here because it could be a PDM, PLM , home >>> brewed python server script to a local or remote DB etc.. >>> References: >>> https://github.com/inventree/InvenTree/discussions/4133 >>> >>> https://docs.google.com/document/d/1qXKEa9Djz1VnEzH7jK0rJ1p9Vu82tq4p_m2zBuNMG6k/edit#heading=h.fyuo1bpfgubj >>> https://gitlab.com/kicad/code/kicad/-/issues/12027 >>> >>> >>> >>> On Tuesday, July 26, 2022 at 1:52:14 AM UTC+8 j...@craftyjon.com wrote: >>> >>>> Yes, the database library would be a type of library format (exact UX >>>> is not yet determined). >>>> >>>> -Jon >>>> >>>> On Thu, Jul 21, 2022 at 7:00 PM Andre Iwers <iwe...@gmail.com> wrote: >>>> >>>>> I assume that this will be some sort of library format and users would >>>>> be able to select this in here right? >>>>> [image: Screenshot 2022-07-22 084207.png] >>>>> >>>>> I would be more than happy to use a non-compiled cross-platform plugin >>>>> system. What's important to me, and what makes my and my colleagues life >>>>> easier, is that it is well integrated and offers a unified interface; >>>>> similar to what I showed here: >>>>> https://gitlab.com/kicad/code/kicad/-/issues/12027 >>>>> >>>>> Apart from this, I am more than happy to use kicad's internal tools to >>>>> create symbols and footprints and place those using the currently >>>>> available >>>>> facilities. In fact, currently our meta data also contains the right >>>>> footprint string which reduced accidental placement of the wrong >>>>> footprints >>>>> a fair bit; it's also a QM thing. >>>>> >>>>> On Thursday, 21 July 2022 at 11:11:48 pm UTC+10 j...@craftyjon.com >>>>> wrote: >>>>> >>>>>> The ODBC interface is designed to support pulling component libraries >>>>>> from the database, not adding metadata to parts that are from other >>>>>> libraries. So it's a slightly different (but related) use case. At the >>>>>> moment there are no plans to support anything other than ODBC. >>>>>> >>>>>> The use case of adding metadata to existing parts that come from >>>>>> other libraries is what you seem to be heading towards. This one I >>>>>> would >>>>>> recommend we look at solving with a cross-platform plugin system of some >>>>>> sort. >>>>>> >>>>>> -Jon >>>>>> >>>>>> On Thu, Jul 21, 2022 at 1:05 AM Andre Iwers <iwe...@gmail.com> wrote: >>>>>> >>>>>>> Hi Jon & Seth, >>>>>>> >>>>>>> Thanks for sharing your thoughts! >>>>>>> >>>>>>> Some thoughts here from my side: >>>>>>> >>>>>>> 1. Storing part metadata in a private database -- I have started >>>>>>> working on this feature (#7436 >>>>>>> <https://gitlab.com/kicad/code/kicad/-/issues/7436>). The scope >>>>>>> of this one is that the database is SQL-like and ODBC-compatible. >>>>>>> - I am not sure if this relying on ODBC is flexible enough. >>>>>>> In my case it would not be. I mostly work with either Navision or >>>>>>> Inventree >>>>>>> which stock data is managed by other people. In my case I rely >>>>>>> heavily on >>>>>>> their stock/metadata and I don't have direct access to the >>>>>>> source. It's >>>>>>> read-only. Having an ODBC compatible interface would mean that >>>>>>> every >>>>>>> software you'd need to work with must support that, or am I wrong >>>>>>> here? >>>>>>> - My approach tried to avoid exactly that. I have a driver which >>>>>>> does all the translation work between kicad and the inventory >>>>>>> system and >>>>>>> the user can design that plugin as needed. Kicad would only >>>>>>> impIement that >>>>>>> once and that's it; there is no need to touch the source again >>>>>>> when >>>>>>> changing the plugins itself. >>>>>>> - I admit that a compiled library might not be ideal as kicad >>>>>>> is cross-platform, however, one could use a simple >>>>>>> inventory_plugin.py file >>>>>>> or archive which is digested by kicad and then use a python >>>>>>> interpreter. >>>>>>> >>>>>>> >>>>>>> 1. Retrieving live part metadata from a public data source >>>>>>> (Mouser/Digi-Key/Octopart/JLC/etc) and applying it to already-placed >>>>>>> symbols (or newly-placed). >>>>>>> - The above approach could be used here too. Digikey provides >>>>>>> API access and everyone can get their own API key. >>>>>>> >>>>>>> >>>>>>> Please let me know what you think. >>>>>>> >>>>>>> Cheers, >>>>>>> Andre >>>>>>> >>>>>>> On Tuesday, 19 July 2022 at 11:47:45 pm UTC+10 se...@kipro-pcb.com >>>>>>> wrote: >>>>>>> >>>>>>>> Just to add on to what Jon has already said: >>>>>>>> >>>>>>>> KiCad is a cross-platform application, so any solution needs to >>>>>>>> similarly be cross-platform. If the target for the external plugin is >>>>>>>> to >>>>>>>> allow users and companies to develop their own interfaces that can be >>>>>>>> used >>>>>>>> in KiCad, then this needs to be done in a scripting language. >>>>>>>> >>>>>>>> There are a couple reasons for this. The big one is that MacOS >>>>>>>> cannot load a binary plugin from outside the signed application >>>>>>>> without >>>>>>>> triggering GateKeeper and all of the badness that comes with that. >>>>>>>> >>>>>>>> We can build/distribute binary plugins (like the 3d plugins) that >>>>>>>> are included in our codebase because these get signed with the rest of >>>>>>>> the >>>>>>>> application. So, if the target is to allow ODBC interaction, we can >>>>>>>> probably support a binary plugin interface that explicitly does that. >>>>>>>> I >>>>>>>> think that this is (#7436 >>>>>>>> <https://gitlab.com/kicad/code/kicad/-/issues/7436>) >>>>>>>> >>>>>>>> Seth >>>>>>>> >>>>>>>> On Tue, Jul 19, 2022 at 6:05 AM Jon Evans <j...@craftyjon.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Andre, >>>>>>>>> >>>>>>>>> I think it would be best to have two different discussions here, >>>>>>>>> first about the idea and concept, and second about your proposed >>>>>>>>> implementation. >>>>>>>>> I wanted to lead with this because I think there is a tendency in >>>>>>>>> discussions to mix the two together unnecessarily. >>>>>>>>> >>>>>>>>> This is a real need and something that we've already started >>>>>>>>> working on related things for, although in a slightly different >>>>>>>>> direction >>>>>>>>> than you describe. There are two different but related concepts >>>>>>>>> here: >>>>>>>>> >>>>>>>>> 1) Storing part metadata in a private database -- I have started >>>>>>>>> working on this feature (#7436 >>>>>>>>> <https://gitlab.com/kicad/code/kicad/-/issues/7436>). The scope >>>>>>>>> of this one is that the database is SQL-like and ODBC-compatible. >>>>>>>>> >>>>>>>>> 2) Retrieving live part metadata from a public data source >>>>>>>>> (Mouser/Digi-Key/Octopart/JLC/etc) and applying it to already-placed >>>>>>>>> symbols (or newly-placed). >>>>>>>>> >>>>>>>>> (1) can obviously be built into KiCad and so it will be. (2) is >>>>>>>>> more challenging -- most of these data providers would be unhappy >>>>>>>>> with >>>>>>>>> KiCad providing an easy way for users to scrape their data, as it >>>>>>>>> could >>>>>>>>> cost them a lot of money. There have been discussions with various >>>>>>>>> data >>>>>>>>> vendors about an integration approach, but in general this is >>>>>>>>> something >>>>>>>>> that needs to be negotiated per-vendor. >>>>>>>>> >>>>>>>>> Because of this, I agree that the right approach is to build a >>>>>>>>> framework rather than an implementation, which leaves e.g. Digi-Key >>>>>>>>> open to >>>>>>>>> providing their own KiCad data plugin if they wish to make their data >>>>>>>>> available to KiCad users in this way. >>>>>>>>> >>>>>>>>> This leads me to the implementation -- I'm not sure that creating >>>>>>>>> a new binary plugin interface is the approach we should take for >>>>>>>>> this. We >>>>>>>>> have been investing in the Python plugin ecosystem elsewhere, and >>>>>>>>> this >>>>>>>>> seems like an ideal fit for Python: people creating datasource >>>>>>>>> plugins can >>>>>>>>> distribute a single plugin that runs on any platform KiCad is >>>>>>>>> supported in, >>>>>>>>> and there are fewer stability and security concerns with loading an >>>>>>>>> arbitrary Python plugin than with an arbitrary compiled binary. >>>>>>>>> >>>>>>>>> For the workflow you describe: this makes sense in the use-case of >>>>>>>>> adding metadata from an external source to parts that were already >>>>>>>>> placed. >>>>>>>>> The Database Libraries workflow will be oriented more about placing >>>>>>>>> symbols >>>>>>>>> with all their metadata from the start. Perhaps it would be possible >>>>>>>>> to >>>>>>>>> have this proposed metadata plugins interface work either at initial >>>>>>>>> placement or afterwards? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> -Jon >>>>>>>>> >>>>>>>>> On Mon, Jul 18, 2022 at 6:53 PM Andre Iwers <iwe...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi There! >>>>>>>>>> >>>>>>>>>> I constantly run into issues where I need to add all sorts of >>>>>>>>>> metadata to my sometimes 100+ parts schematics which are saved in an >>>>>>>>>> external inventory system. Initially doing it by hand I grew tired >>>>>>>>>> of it >>>>>>>>>> and decided to whip up a slightly less laborious way using a plugin >>>>>>>>>> approach which is quite flexible in terms of datasources. Also >>>>>>>>>> updating >>>>>>>>>> those is even worse. >>>>>>>>>> >>>>>>>>>> I was interested if this is something Kicad would be interested >>>>>>>>>> in taking over for the broader public. I hooked it into the 'Symbol >>>>>>>>>> Properties' or 'Symbol Fields Table' dialog for ease of use and >>>>>>>>>> touched >>>>>>>>>> that existing code only marginally. >>>>>>>>>> >>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_expected-behavior>Quick >>>>>>>>>> >>>>>>>>>> Overview: >>>>>>>>>> >>>>>>>>>> When I have selected a schematic symbol from the standard >>>>>>>>>> library, I'd like to be able to open the 'Symbol Properties' or >>>>>>>>>> 'Symbol >>>>>>>>>> Fields Table' to fetch part information from an external source. >>>>>>>>>> >>>>>>>>>> Idea: Eeschema would simply provide a framework which connects to >>>>>>>>>> one or more plugins and the plugins themselves would be provided by >>>>>>>>>> the >>>>>>>>>> community as .dll or .so which then are saved somewhere on the >>>>>>>>>> computer and >>>>>>>>>> simply loaded when needed. The plugin can be programmed in any >>>>>>>>>> language >>>>>>>>>> such as python, java, c# etc. and would only need to honour the >>>>>>>>>> defined C >>>>>>>>>> wrapper interface. Plugins which don't adhere to that are simply >>>>>>>>>> ignored. >>>>>>>>>> >>>>>>>>>> I have created a first draft of that framework to solve a >>>>>>>>>> laborious task I face once in a while and am wondering if this is >>>>>>>>>> something >>>>>>>>>> that is generally useful for the broader public. I also programmed a >>>>>>>>>> plugin >>>>>>>>>> for Inventree which is used here as part database. >>>>>>>>>> >>>>>>>>>> In general: the framework is intended to use multiple plugins at >>>>>>>>>> the same time so that a user could potentially query for example >>>>>>>>>> Mouser, >>>>>>>>>> Digikey, Navision and Inventree at the same time. >>>>>>>>>> >>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_the-workflow>The >>>>>>>>>> >>>>>>>>>> workflow: >>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_a-brand-new-project-would-be-to>A >>>>>>>>>> >>>>>>>>>> brand new project would be to: >>>>>>>>>> >>>>>>>>>> - select a symbol from the standard kicad library >>>>>>>>>> - open plugin search and search for the specific part >>>>>>>>>> - transfer all metadata automatically into your schematic >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_existing-project>Existing >>>>>>>>>> >>>>>>>>>> project: >>>>>>>>>> >>>>>>>>>> - Open fields editor and bulk apply metadata using the above >>>>>>>>>> mentioned approach. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "KiCad Developers" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to devlist+u...@kicad.org. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/a/kicad.org/d/msgid/devlist/CABQVKmKZAQtXZ-gdQOJiHGH_vJmwfU5Ho%3DCAaMtM_YD2PbZteQ%40mail.gmail.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/a/kicad.org/d/msgid/devlist/CABQVKmKZAQtXZ-gdQOJiHGH_vJmwfU5Ho%3DCAaMtM_YD2PbZteQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "KiCad Developers" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to devlist+u...@kicad.org. >>>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/a/kicad.org/d/msgid/devlist/CA%2BqGbCBOVz_8CJLG5spNgg%3D%3D6RF4s390abE_LW8wQeeE5-A%3DdQ%40mail.gmail.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/a/kicad.org/d/msgid/devlist/CA%2BqGbCBOVz_8CJLG5spNgg%3D%3D6RF4s390abE_LW8wQeeE5-A%3DdQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> [image: KiCad Services Corporation Logo] >>>>>>>> Seth Hillbrand >>>>>>>> *Lead Developer* >>>>>>>> +1-530-302-5483 <(530)%20302-5483> >>>>>>>> Long Beach, CA >>>>>>>> www.kipro-pcb.com in...@kipro-pcb.com >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "KiCad Developers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to devlist+u...@kicad.org. >>>>>>> >>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/a/kicad.org/d/msgid/devlist/2838e643-a03c-4cdb-8853-3228bfbd19d0n%40kicad.org >>>>>>> >>>>>>> <https://groups.google.com/a/kicad.org/d/msgid/devlist/2838e643-a03c-4cdb-8853-3228bfbd19d0n%40kicad.org?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "KiCad Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to devlist+unsubscr...@kicad.org. To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/cfe76e8b-0095-4efb-b10e-248a731a6ae0n%40kicad.org.
