The description 'DER' is ambiguous, but in the contrast you quoted it probably means that RSA keys are in the ASN.1 format(s?) defined by PKCS1 aka RFC8017 et pred Appendix A.1 which like all ASN.1 data can be and for crypto often is encoded in DER. 'X.963' is clearly a mistake; the relevant standards for ECC crypto are X9.62 and X9.63, but the publickey format which indeed is not ASN.1/DER was defined by X9.62 and copied by X9.63, while the privatekey format was not standardized at all, but is often a raw octet string and not ASN.1/DER. (X.(number) standards are from the international treaty organization formerly called International Consultative Committee on Telephony and Telegraphy CCITT and now International Telecommunication Union Telecommunication Standardization Sector ITU-T; X9.(number) are from the USA private-sector financial-industry organization ANSI accredited standards committee (ASC) X9.) OTOH Java crypto encoding for private key is PKCS8 aka RFC5208 (unencrypted) which adds metadata. To convert a Java key to the format your 'wolfcrypt' apparently wants is fairly easy, by just extracting the algorithm-dependent element, while the reverse, using a 'wolfcrypt' key in Java, would be quite difficult with just Java, but adding BouncyCastle helps a lot; there are numerous existing Qs about both (or all three) of these, which I will dig up for you later.
Some other, partial, comments:
generateChaChaPoly1305Key, generateAESGCMKey: KeyGenerator.init(int) takes the number of bits, which should be 256 for ChaCha20 and 128, 192 or 256 for AES. (So does KeyPairGenerator and you got that one right for RSA.) The IV should NOT be generated with the key for AES-GCM; see more below.
unpackRsaPrivateKey, unpackRsaPublicKey, unpackECCPrivateKey, unpackECCPublicKey: the 'PrivateKey' methods use key.getPublicKey() and the 'PublicKey' methods use key.getPrivateKey() which appears to be backwards. Assuming you change the PrivateKey methods to use the privatekey, as above the Java encoding of a privatekey is PKCS8EncodedKeySpec -- not X509EncodedKeySpec which is only for publickey.
unpackAESGCMKey, unpackChaChaPoly1305Key: to use a whole byte[] as the key, you don't need to specify , 0, array.length, you can use the simpler ctor SecretKeySpec(byte[], String algorithm).
encrypt*: you do new String(byte[]) on the ciphertext, and (String_var).getBytes() in the decrypt* methods. THIS IS NOT SAFE. Java String is designed to hold characters, and although crypto still uses the traditional terms plaintext and ciphertext, since about 1950 the plaintext is not required to be and the ciphertext is NEVER actually characters, but rather arbitrary bit patterns. Trying to put these bits in a Java String and then get them back will usually fail, especially if done across multiple systems (e.g. encrypt on A, transmit, and decrypt on B, which is a common use case). The best thing is to handle them consistently as byte[]; if you must run them through something that can't handle arbitrary bits, like SMS or the Web (HTML), you need to encode in a text form that preserves all bits; the common methods for this are hexadecimal (or hex) and base64.
encryptAESGCM: it's not clear what key.getIV() does/uses, but if this repeats an IV value for multiple encryptions using the same key, that is CATASTROPHIC; even one reuse destroys all authenticity (attacker can forge any data), and depending on your data anywhere from one to a small number of reuses (like two or ten) destroys confidentiality (attacker can expose supposedly secret data).
encryptChaChaPoly1305: this clearly uses the same nonce, all-zeros, for every encryption; if you reuse the key at all, which seems to be the point of your design, this destroys all security in the same way as for AES-GCM.
At this point I gave up. You should throw out this design and replace it with one from someone who knows about cryptography.
