Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files that game developers need to collaborate on. For example, one artist might need to obtain an exclusive lock on some art assets while editing them, because there is no sane way to merge two artists' async edits.
The SOTA in this area is Perforce (https://www.perforce.com/products/helix-core), a proprietary system. From what my gamedev friends tell me, when Perforce works it's great, but it hits enough snags that you need a tools engineer to manage it and occasionally fix issues manually. Git LFS is an alternative, but my gamedev friends all prefer Perforce especially when working on team projects beyond like 3-4 people.
This was within a bank and the code in question was related to enabling Apple Pay from within the banking application. The consequences of that information and code leaking or being seen by anyone who had not signed the NDA were very serious (don't remember the details but it made the lawyers were extremely stressed about it).
Needing to figure out a way to protect those parts of the codebase it was decided in the end that the "easiest" way of doing this was to split the repository in half, with the actual artifact building taking place from the half that had the NDA code. The rest of the application (basically the whole application) was then used as a dependency by it.
Still didn't quite solve the issue, but access to that repository was heavily controlled.
Keys, secrets, etc. yes. But code? What am I missing here?
See also: the time that ATI's CEO told his employees that their chips would be powering Apple's to-be-announced hardware a few days before the announcement. Steve Jobs responded by pulling all of ATI's hardware from its demo units at the announcement, not mentioning ATI at all, cancelling a joint demonstration of the Radeon card that was going to be in the system, and never partnering with ATI again.
https://web.archive.org/web/20001216031800/https://www.zdnet...
That is partially incorrect; you can restrict writes via hooks but not reads; you'd need a workaround like submodules
It’s fine for things that you want devs to be able to see without the Git host being able to see them, it’s less good at RBAC because there’s no real “identity” component at read-time.
Git enthusiast would often tell you to do this separately with a submodule, and set permission on the version control forge software level (which means Gitea/Github private RBAC access to certain repos for cloning), sure, but that is also painful as hell.
But my point is that all of this is exactly by design from Linus Torvalds's need for Linux Kernel to replace BitKeeper. Git simply isn't the tool for everything, it was developed for a software project with liberalism in mind, but corporate stuff is monoculture and prefers proprietary, shut-in model, and the eat your own dog food mindset, and no wonder it is so painful to deal with.
Unfortunately there are many people who think git is a panacea and is suitable for all version control tasks of anything.
In fairness, p4 predates most of the VCSes we consider "modern", so I empathize with a lot of the underlying architecture decisions. However, it has and continues to utterly fail at improving at a reasonable pace.
For example:
- p4 tracks file metadata of client workspaces on the server (sync'ed locally, opened for edit, file revision, etc) and uses this as the basis to avoid doing unneeded work. If this becomes desync'ed, a reconcile or force sync must be used. A reconcile can take hours, potentially days; it tries do detect file moves by default, so likely at least O(c^n) for some c>1. I have never personally seen a default reconcile operation _complete_ over any modestly large game code base, and in practice, people accumulate a litany of workarounds and scripts to fix this for themselves.
- Scripting p4 is a nightmare. Documentation is poor, schemas do not exist, and all the language-specific libraries are just thin wrappers over its C++ API.
- By default, p4 "helps" you with text files by "correcting" line endings on sync or even converting between encodings. This works until you have a mixed-OS environment, and discover a part of the pipechain that _must_ have a certain style. There are various levers to pull to make this better, but I've yet to find something fool proof.
- By default, p4 keeps flies read-only, only unlocking them when explicitly marked as being edited. This means, to avoid having to do this manually, every tool you use needs to be p4-aware. Or, you can turn this off, and choose to contend reconcile instead. (See above)
- Branching a modest game project, with, say, Unreal source code, can take hours. And this is the quick version where you ask the server to simply create new metadata, with no file transfer to a client.
- p4 is licensed by the user-account. Every user entity in p4 not intended exclusively for performing backups and maintenance operations counts toward this, including users required to integrate with other services. Plus, often times, these integration users must have admin access to be useful. The security posture is horrific.If perforce is the best there is out there for large binary asset management, then there is a blue ocean worth of potential improvement for git.
Perforce is a piece of crap, a relic of the 20th century that must die in a fiery inferno.
Thats not how everyone else works.
We're all using package managers to help with massive amounts of 3rd party dependancies (why are you version pinning in any place other than your repo, why arent you pulling updates through your repo and reviewing them)
We're reliant on tools like artifactory to make sure those depedancys dont disappear or are not corrupted.
We use yet other tools to manage our binary files (this tool would fix that).
Github, gittea, gitlab, bitbucket... have all added piles of tooling around git, that are grafted on around its short comings.
> It's crazy to me that nobody has made anything better yet.
Because our entire industry has fallen into the rut of "more tools", of stacking turtles (https://en.wikipedia.org/wiki/Turtles_all_the_way_down ) rather than fixing the real issues that hold us back.
> Although I could start that myself and yet have not.
Because unless your a Google or a Linus, no one is going to look twice at your tool for something that is this important. Im not even sure that epic games has the good will, or trust to launch this.
I am going to give them the benefit of the doubt and take a long hard look at it, but my optimism is tempered. But unless it offers a LOT more than git, the extra overhead (lacking IDE support, deployment changes and all the other tooling in GIT's orbit) it isnt going to be a worth while change.
He’s in a category of influencers who post constantly about gripes and grievances and smug superiority. Some people like that content but I can’t stand it.
I really like hearing about indie development and small teams, but you don’t have to present everything as condescending superiority over the industry. That’s not the part I find interesting.
I think being influential just does that to people, with high regularity.
Huge teams are more often than not the sign of bloat and inefficiences.
Now there's an argument to be made that many don't need the remaining ones but to claim that you 'replaced PowerPoint' for anyone but yourself is ridiculous.
They're good at demos, I give them that.
https://epicgames.github.io/lore/explanation/system-design/#...
Also git-lfs is a crutch that breaks more often than it works :/
(I agree though that for small game projects, git is mostly 'good enough', even without lfs).
For the longest time Git tools were really poor. In recent years there's a few ok ones, like Git Fork, though I wouldn't know if those tools scale to the level of a AAA team size repo and not fall over.
it's also tough when you have 1TB of data, over 1mm files and you might want to lock hundreds files in one go
>Lore, formerly called Unreal Revision Control, is the built-in version control system for UEFN (Unreal Editor for Fortnite), where creators have been using it to version their islands. It is also seeing progressive adoption by internal Epic teams, and is being implemented as the backing store for UEFN’s cook pipeline, where it replaces traditional intermediary storage layers—eliminating redundant file transfers and significantly reducing the time between publishing changes and those changes being playable.
Surprised it is in Rust and not Epic C++ or Verse. I wonder why.
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 10 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 290 bytes | 290.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
From the documentation, it looks like Lore does a bit better in this regard:
Pushing 1 fragment(s)
Pushed 1 fragment(s), 124.00 bytes
Pushing a3f8c2d1... to branch main
Pushed revision 1 -> a3f8c2d1... to branch mainThe objects are referenced by trees. A tree is just a directory.
The trees are then referenced by commits and/or tags into a DAG with named pointers into various parts of it (which are your branch and tag references):
https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
Because it would be terribly in-efficient to have a bunch of loose objects, git periodically groups them together into packs. To save space, the objects are compressed against one another (delta compression) within the packs.
https://git-scm.com/docs/git-pack-objects
https://github.com/git/git/blob/master/Documentation/technic...
When pushing or pulling, the git transfer protocol basically enumerates what objects each side has so that it only needs to transfer the difference. On top of that, it delta compresses the objects on each side that aren't already grouped into packs against each other to save space.
https://github.com/git/git/blob/master/Documentation/technic...
Because git is an open-source project written by nerds, it shows you all of this information. Feel free to ignore it!
But if you really want to know, it's all documented both in the git book and git documentation directory, both linked above.
(Caveat: I'm working from memory and surely got some detail at least slightly wrong.)
There are 1000 things that's true of about git. At a certain point that becomes a problem in and of itself.
The git support for new employees drops basically to zero.
I’d be a bit worried if git didn’t heave that particular contented sigh when I ask it to push
Not saying Lore's approach is bad, but sometimes "worse is better".
Still the porcelain is more like cold stainless steel
Perforce definitely needs a challenger. It is not the incumbent because it is particularily simple to use or administer. Git is actually way simpler when it comes to branching operations for example.
The reasons why p4 is often preferred in gamedev have already been mentioned in other comments: large project support, permissions, file locking and so on. Another key reason p4 is the king for Unreal dev is just how well it's supported inside the engine. Not perfect, but it's the best supported VCS because it's what Epic uses. Even the Git plugin is painfully unfinished, because Epic does not internally use it. So with Lore I expect them to give it first class support. I'd recommend Git a lot more if the support in Unreal was better.
(background; I've been in gamedev for almost two decades now, 2-200 person companies, every kind of engine and version control system. I prefer git where I can use it: for Unreal that means small projects and/or tech savvy team members. Pick the tool that is right for the job and the team.)
I'm not just picking nits here. And this is not cynicism.
so there you go.
[0] https://en.wikipedia.org/wiki/List_of_minor_Star_Trek:_The_N...
It's more likely that they took advantage of some things in the license that you weren't aware of, since stealing would be illegal.
- Pachyderm (Go): https://github.com/pachyderm/pachyderm
- XetHub (acquired by HuggingFace): https://huggingface.co/blog/xethub-joins-hf
- LakeFS (Go): https://github.com/treeverse/lakeFS
- Oxen (Rust): https://github.com/Oxen-AI/Oxen
I guess with AI, anyone can vibe code a content-addressed, chunk-level deduped, versioning system in Rust these days...
But jokes aside, Lore seems really cool! What's interesting is the realization that different domains/industries have similar problems, but they don't seem to be cross-polinating. In this case AI and Gaming both need a storage system that can version control large binary files at scale. I think there's lots of opportunities to share ideas here, but perhaps the lack of idea sharing (currently) creates opportunity!
That already warrants different storage architectures.
[1]: https://gavinhoward.com/uploads/designs/yore.md
[2]: My WIP VCS has been named Yore for at least two years; I did not copy Lore's name.
It also makes diffing hard, as well as diff viewing.
> Open-source the desktop client so the community can build on its full graphical experience, not just download it. An early desktop client already exists as a binary download, but it isn't open source yet — it depends on some proprietary components, including Epic's internal design system. We're working to make all of it available in the open so that the client can ship as source alongside the rest of Lore. Lore is an open project, so it is important that the desktop client — which will be one of the main ways many people will interact with Lore — is also fully open so that the community is free to review, extend, and shape it.
What a shame
they do say they will open source it, but who knows:
"It isn’t open source yet—it currently depends on some proprietary components, including Epic’s internal design system—but we’re committed to open-sourcing it in the future"
I expect fully to mean fully, though.
Passing for now.
Looks very git-ish. But probably better equipped for large binary files.
echo "Hello, Lore" > hello.txt
lore stage hello.txt
lore status --scan
lore commit "Initial revision"
lore pushIf the roadmap's "Web client and code review tools" could replace gerrit for me, this would be a easy switch.
Moreover, it looks like they designed both the mutable store and immutable store to be able to easily store their state directly on an s3 like system.
There are a number of features that would greatly speed up CI/CD system operations I belive.
EDIT: and then an example for the merge stuff I couldn't find while typing before: https://mergiraf.org/ and HN discussion a few years ago: https://news.ycombinator.com/item?id=42093756
[0] https://github.com/EpicGames/lore/tree/main/docs/developing/...
[1] https://github.com/EpicGames/lore/blob/main/docs/developing/...
I'm trying to figure out what Lore can accomplish that git+LFS can't. I've read about big binaries chunking, native interface and permission, is there anything else? Weren't those problems already solvable in the git+LFS ecosystem?
Count on it.
From the horses mouth itself:
https://x.com/timsweeneyepic/status/964284402741149698?lang=...
He straight up makes fun of Linux because he hates Valve and GabeN.
https://x.com/TimSweeneyEpic/status/2054680825677910439?s=20
It is interesting that people are so cynical about Epic giving out free games. I get that people love Steam, but competition in the storefront market is not bad.
But Kojima games require me to be in a very specific mood and I have been trying to work my way up to it.
Lore is meant for situations where your repository is going to contain gigabytes of binary files, such as art assets for games. Git is technically great at everything but that, and even the external solutions for that situation still kind of suck.
Its remarkable that anyone thought this website was fit for release, and it gives off strong slop vibes
I also have absolutely zero trust in a product like version control being provided by a for-profit company. It seems like a terrible idea to tie your software stack to Epic Games of all people, given their track record
It’s like anything you do has to talk to the server
Even something as simple as diffing a file will just hang if there are server issues
Anyway it's probably fine software and I am genuinely going to give it a shot for a usecase I have involving large image files. But the LLM-generated docs don't inspire confidence.
[1] https://epicgames.github.io/lore/explanation/system-design/
[2] They literally have a section header "10.1 Revision state as a 320-byte fragment". The byte size isn't even relevant in the code as an implementation detail, much less belongs in a design doc. No one read this doc before publishing it.
On the flip side, I expect the project itself will be workable - well, assuming they're actually using it themselves! UE is a big pile of Stuff on its own, and Fortnite must have god knows how much additional crap in there, so if this is (or will be) their replacement for Perforce internally, then it'll be getting a good deal of testing. (If they're just chucking it over the wall, though... well, sheesh, you first...)
(Perforce is the standard thing for games, and pretty well it works too, and hopefully this will deliver it a well-needed kick. It was sold to private equity about 10 years ago, and it feels like they've been coasting ever since. (Perhaps users in other sectors are happier though?))
> 3.2 Explicit non-goals¶
> Peer-to-peer decentralization. Lore is centralized by design. Two clients communicate through the remote, not directly.