As you make your dataset bigger, then based on some local unit of data such as 4 or 100 floating-point numbers (which are internally rather linear, such as stacks of layers and internals of the most local atom of data, which both provide linear correspondencies you often overlook by O-notation philosophy unless for very particular small numbers, where these rules break in material realms - the O-notation is kind of heavenly, as it applies rather for statistical approximation in long term, and many exceptions could be found where strict constraints for local cases are known or come in sets with low local change - then, you just calculate the precise number of operations in simplest case, and compare them basis-to-basis).
Exponential complexity:
- Data items are trivial to measure in tokens, but tokens are roughly linear to letters - you can find very linear average number on statistical propagation.
- Real AI operation operates on data items in manner, which grows the implication cost in square fashion in relation to this linear fashion - for each added item, the complexity or time and effort linear (anti)performance will change in imponential or exponential relation. 2 elements have complexity roughly 4, but adding 1 - 3 elements have it already 9, where 4 out of this 5 appeared from this exponent and not mere combination.
This complexity, indeed, is very fruitful and not undesirable:
- In exponent space, the elements of complexity do not yield particular incomes, but an efficiency growth in accelerated space.
- If we are aware and focused on each two-element pair in two-vector combination, where combinations appear between past and next moment in all possible pairs made of these vectors - one for future, and one for past layer where the learning matrix is actually in between. So we move a little outside of time and space, into metaphysical "eternality-projection", and we contemplate what it means to "combine" in this space where future is born, or not yet known, and past is given.
- Attention and self-attention contemplate on all combinations of matrix, again combining two vectors - for self-attention, the vectors are the same, and for other-attention, they are still in same imaginary or metatime event, at the same moment in math space where implication is also a change in the moment, where we know already calculated things, but the future kind of resolves more logic: we are in this pragmatic space of optimization, which is not merely perfect logical space but where we do hard work to achieve results.
- This is very funny: if you have x length of input vector, and x length of output vector, then without looking at any internal, hidden layers, the matrix can be used to directly combine all the possible combinations between past and future of the same thing - a mindful substance which then works on functional, logical space rather than in imperative or algorithm: in this more logic-close theory, we can think that implication or calculation transforms input element into the output, which is more close to equations and logic math thinking, where thinking the question is answered is more close to our intuition about world - even if we do a math implication, which correlates the axes of these paradigms (functional-logical one without side-effect, and then the imperative way of programming algorithm - a very unstable system which is not like metalogic, but then wow it has the physical laws we can use intuitively without much thought - the understanding of time and communication comes naturally and is the materialist guide to this, from practice rather than theory, into imperative and extrospective rather than contemplative and introspective, where question transforms to answer through an information equation we can measure and see).
Summarization of elements into APIs is one way to control complexity:
- Summary is so much shorter, with complexity reduced so much, that we can add many summaries into a document.
- In this document, we relate those summaries based on local aspects and general item for fine-tuning.
- Those summaries inside, are matrix-calculated siding each to each.
I assume that the operation is input => output, Q => A. To match the outside pattern of using them, and inside pattern of using the tokens, we can relate each input to each output - and it's even harder to construct sets with 3 or more variables, but they appear if we think we measure rather dimensions than particulars: if these are three dimensions in the vector and matrix, where it becomes tensor and weights and biases, something more high-level conceptualized, we can see in 3 dimensions of binary pairs, strict order of numbers into + and - rather than any direction, we find 8 solutions which determine the space where we can optimize.
You need to set up linear operations:
- You write rules to process the cards.
- You bring the results to other cards.
- You are aware of input and output target cards, where you define what will be transferred: visible for target as context, for example, but not to set a single rule (you can use Prolog to create complex, standard associations: use SpaCy to convert natural language expressions to Prolog, and find subsets of logic spaces where you can relate a few relations; do not build huge realms where all is known at one - it will be lost in fact travel probabilities, again if you are not IBM or Microsoft).
If you have set rules, which determine that single element in input vector, or sets of elements, which come in small number, gives certain sources to small number of elements in other vector: this process is very linear and if AI processes single nodes, it is more linear. You can turn attention to single sources, where the target where information reaches, forms it's references: users of it's fact and setters of it's developed-evolved basis.
Markdown-manual processing: you summarize each of your file, you copy short summary to reference file to make up the source, you add the link for further processing by AI in this context, and you assign localization, such as details contextually relative about source document, to finish this process.
- AI can learn this manual processing and automate it based on examples, and even non-specialized AI can do this with some manual work, where CoPilot is your genuine assistant again.
Where you have well-known rules, you can define prolog-like rules or even rules in simpler language, where you can only do implication and check for it. You need to be creative with this language, and seduce an AI to create consistent structures rather than you trying to be logic-strict, because we ain't in that kind of world where this static logic matters.
Finally, you reduce complexity by enabling proper refactoring.
Refactoring means:
- You collect and store the AI commands and process flow you used to generate your code.
- You document the exact code you got, but this documentation might be different for some regenerations: you can use it as a basis for new generation or version, or make sure you know what you updates.
Now, to enable refactoring:
- You create basic task, you make sure it's not particular, but it also includes how you reason about the context and scope, as well as timely progress.
- You make sure this task, and documentation of generation, is available with your API, which is the major bottleneck through which you enter your body or spec and which is short: the code itself is not what you rely on, but you set quality measures and standards each time you see better code for this, and you can specify it's not less with that best code, with no unreasonable changes unless it's the goal.
To really make your product part of this world, you create these "librarity" elements as I call this:
- You change it in different directions, and create instructions, comparisons and generalizations, as well as first-hand instructions to generate this.
- Based on having those and more fields (instructions, comparisons, results), you can already autogenerate large number of cards: to have general-purpose AI, you do not feed these all at once; rather, before you become overspecialized, you already repeat all the experiments or iterations with similar dataset, where the AI won't specialize even if you give it enough to substantially re-learn as if it's final training still provided random probability input: you keep tensors also on your past goals to not make it specialized or forget; why it does not forget is too complex magic, but it's sure how the local samples interact rather than past and current input having two-way, strong interaction so sure to form interactive, holistic patterns - the latter is not trivial from form of Perceptron, altough it has some such effects for sure but only because it's "smart"? :)
- You create systems where you incorporate this, and give instructions.
- You are agile programmer, an agile AI user, who wants small samples and examples, and works on his personal database.
Then, you all register all your cards in common server and train one AI: it starts to reason. Share the formats, needs, sources and things you are interested in, to get patterns and specifics - or make a chain of an AI, where everybody does prework like tokenizing, and models are trained by somebody to next phase and given over: here, linearization is problem rather than solution, because it's hard to fully parallelize and fully use not only time, but the space - space is many computers of many users.