'Augmented' Reality with OpenPeripherals

Because I can't focus on anything for more than 2-3 days, I present to you: OpenPeripherals augmented reality:



Slightly longer video

If you can't tell what is happening: that red point and box are being translated so they are always over the sensor. This uses OpenPeripheral's terminal glasses to draw the screen and its sensor to get where I am looking and my position. I'm then converting that information into a matrix and projecting the point to it renders in the correct location. This could easily be extended to draw lines between points or entire objects in world.

However, there are several many caveats:

• High latency: getting player position takes a tick, then sending to the client takes another tick so there is a large delay between user movement and what is show.
• No Z buffer/Z-sorting: so objects closer to you may not be drawn in front of objects further away.
• Things aren't perfectly positioned when further away.

If people are interested in the code, it is on GitHub.

I'd say most all of your "caveats" are really just "unimplemented features that need moar code", with the exception of the lag… which isn't that bad. Has a sort of charm to it, really.

Puts me in mind of the cyber interface from System Shock, in particular how it acts before you turn the help system off.

This is also depending on FOV, isn't it? Since the world rendering is depending on that, and the grid system of terminal glasses aren't, right?

EDIT: May as well be one of the bigger sources why it's inaccurate for longer distances

Bomb Bloke, on 22 May 2016 - 03:46 PM said:

I'd say most all of your "caveats" are really just "unimplemented features that need moar code", with the exception of the lag… which isn't that bad. Has a sort of charm to it, really.

True. "Current issues" might be a better title.

InDieTasten, on 22 May 2016 - 05:27 PM said:

This is also depending on FOV, isn't it? Since the world rendering is depending on that, and the grid system of terminal glasses aren't, right?

This depends on FOV, render distance and screen size (though I doubt render distance has an effect). However there are config options for all of these so it should be easy to tweak.

Neato. Make it so it renders a computercraft terminal onto a giant wooden board!

I've integrated my clipping algorithm into this. Lines now work, and things don't render behind you!

It doesn't work for filled triangles yet (something is broken somewhere).

LDDestroier, on 22 May 2016 - 08:43 PM said:

Neato. Make it so it renders a computercraft terminal onto a giant wooden board!

That'd be awesome. Seriously, do it.

This is definitely a very interesting concept and shows potential, keep it up!

This is so cool! It's a shame though about the delay :/

LDDestroier, on 22 May 2016 - 08:43 PM said:

Neato. Make it so it renders a computercraft terminal onto a giant wooden board!

And yeah this would be awesome!

I hacked together a horrible clipping algorithm for triangles. Which means all shapes now work!



There are some issues with triangles vanishing for no reason though not sure why. Fixed! I'm sorting triangles by their distance from the player though again this isn't perfect. Do not the nice animated triangle by the computer though. Well, I think it looks nice anyway :)/>.

LDDestroier, on 22 May 2016 - 08:43 PM said:

Neato. Make it so it renders a computercraft terminal onto a giant wooden board!

It would be cool, but I'm not sure it is worth the effort. Also you'd have to split each character into pixels and then draw them: there are just under 1000 characters on a normal screen. given the number of pixels per character you're looking at 10k polygons: I'm not sure this engine can handle it. There are probably ways to optimise it so there are less polygons but…

Will it work with normal text? If I wanted to have a string hover a location, could I do it with just the addText function rather than drawing each pixel?

I fixed the clipping algorithm (again) so everything is even less buggy. I wrote a silly little "game" where you pick up the floating cube.



It looks better when your playing it. Sort of.

Selim, on 24 May 2016 - 03:29 PM said:

Will it work with normal text? If I wanted to have a string hover a location, could I do it with just the addText function rather than drawing each pixel?

Yeah. It would be kinda cool to use this to add overlays with labels to tiles and entities to give additional information (such as entity health).

For your 'game' perhaps try adding an arrow on screen when the cube isn't on screen showing you where it is.

I had a similar program to this on another language/game and I remember being able to ignore screen size and convert the 2D space to 3D space regardless of graphic settings in a small quick math problem….Honestly no clue if it will port over 100% to minecraft but assuming as long as players keep the same FOV it should work I assume. Give me a minute to download the things I need just to recover this old code


Here you go like I said I don't know how well this will port over but it only needs the FOV modifier instead of screen size (it just continues to follow off screen)

float FOV = 1.7320508075688774;
vector OffScreen = <-1000.0, -1000.0, -1000.0>;
vector Region2HUD(vector objectPos, vector cameraPos, rotation cameraRot)
{
    objectPos = (objectPos - cameraPos)* (ZERO_ROTATION / cameraRot);
    objectPos = <-objectPos.y, objectPos.z, objectPos.x>;
    float xHUD = (objectPos.x * FOV) / objectPos.z;
    if (xHUD > -3.0 &amp;&amp; xHUD < 3.0)
    {
	    float yHUD = (objectPos.y * FOV) / objectPos.z;
	    if (yHUD > -1.0 &amp;&amp; yHUD < 1.0)
	    {
		    float zHUD = (objectPos.z - 2) / objectPos.z;
		    if (zHUD > -1.0 &amp;&amp; zHUD < 1.0)
			    return <0.0, -xHUD / 2.0, yHUD / 2.0>;
	    }
    }
    return OffScreen;
}

Sorry for the necro. I saw this on my project list and decided to do some polishing up. Instead I added another program: I present to you furnace cam!



It doesn't do anything especially fancy: just renders a progress bar of each furnace. I thought it was quite cool having dynamic information about in game items such as objects. Techinically this could be expanded over a whole base. Another thing I'd like to see is applying this to entities: you could provide a health display of nearby mobs.

The source can be found on GitHub.

Edit
I re-implemented Neat using this.

It doesn't change the colour of the bar when it gets low, nor does it render potion effects. But hey, it looks quite good.

This is too good to not necro. As far as the lag is concerned, couldn't you do something similar to what multiplayer games do to make up for the delay? Also, is there a forcefield peripheral that can generate/use another mod to generate invisible forcefields? If so, then you could easily combine the two for some awesome functionality. Another thing I noticed is that your examples, and the functionality possibilities, are very similar to the pneumatic helmet from PneumaticCraft. This is fantastic, as the pneumatic helmet has no ComputerCraft support, so a CC-supported replacement is something I've always wanted. So, for instance, you could create ghost blueprints just using your eyes and then use drones to clear them out.

In combination with Plethora's neural interface and kinetic module, you could turn a second player (second account?) into your personal Terminator with a full readout.

meeko011, on 18 March 2017 - 12:03 PM said:

This is too good to not necro.

Thanks!

meeko011, on 18 March 2017 - 12:03 PM said:

As far as the lag is concerned, couldn't you do something similar to what multiplayer games do to make up for the delay?

Sadly I doubt this would help. In multiplayer games, the rendering is done by the client, with the server sending updates about position and what not. Here, everything is done by the server: user input and rendering, then sent off to the client.You might be able to guess where the user will look next and use that instead, but then you'd get a weird swaying motion if they stop moving suddenly. I guess I could try it…

meeko011, on 18 March 2017 - 12:03 PM said:

Also, is there a forcefield peripheral that can generate/use another mod to generate invisible forcefields? If so, then you could easily combine the two for some awesome functionality.

Not that I'm aware of. It would probably be possible to add compat with MFFS to Plethora, I'll look into it.

meeko011, on 18 March 2017 - 12:03 PM said:

So, for instance, you could create ghost blueprints just using your eyes and then use drones to clear them out.

There's an OpenComputers mod which adds this functionality. I'm planning on adding something similar to Plethora.

This is quite neat, augmented virtual reality :P/>

What you could do is make the server a command computer so then you can add special effects to the augmentation, say, if you touched an augmented object that would usually be harmful you can damage yourself with a command. E.g. spawn a harming potion on yourself or something like that.

EVEN BETTER!

Augmented mobs - Skeleton! An augmented skeleton object which can shoot real arrows at you (using the spawn entity - arrow command). Hardest part would be working out the direction and speed of the arrow to hit you…A fireball might be easier. :P/>

But the ideas are there. You could do all kinds of things with commands stitched into this…*Looking through the commands list in awe of what you could do* :OO

/USE PLETHORAS GLASSES/

AyScorch, on 17 June 2018 - 02:10 PM said:

/USE PLETHORAS GLASSES/

Plethora didn't even exist when I started this (first commit was June 14th 2016), let alone including the overlay glasses! It should be pretty trivial to port this to Plethora, though native 3D support is on the roadmap for Plethora (though has been for several years :/).